Merge pull request #15 from inbo/odbc

stijnvanhoey · web-flow · commit b2b3c48b6bb3 · 2017-03-10T11:11:38.000+01:00
Odbc
diff --git a/docs/data-handling-database-R.html b/docs/data-handling-database-R.html
diff --git a/docs/data-handling-googlesheet-R.html b/docs/data-handling-googlesheet-R.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
@@ -274,23 +280,24 @@ <h2>Setup</h2>
   # save the token into a file in the current directory
   saveRDS(token, file = &quot;googlesheets_token.rds&quot;)
 }</code></pre>
+<pre><code>## Auto-refreshing stale OAuth token.</code></pre>
 <p>Make sure the token is savely stored within your project folder without sharing it or putting it into the version control history. If you need more power (e.g. necessity of integration services such as Travis CI), check the encryption options in <a href="https://rawgit.com/jennybc/googlesheets/master/vignettes/managing-auth-tokens.html">this manual</a>.</p>
 <p>Once registered, an overview of your available google sheets is provided as follows:</p>
 <pre class="r"><code>gs_ls()</code></pre>
-<pre><code>## # A tibble: 72 × 10
+<pre><code>## # A tibble: 167 × 10
 ##                 sheet_title        author  perm version
 ##                       &lt;chr&gt;         &lt;chr&gt; &lt;chr&gt;   &lt;chr&gt;
-## 1  Kopie van Verdeling_kre… liesel.vande…    rw     old
-## 2  Lessenreeks Wetenschapp…     dirk.maes    rw     old
-## 3  IT-klankbordgroep_actie… daniel.duseu…     r     old
-## 4  Buitenlandse zendingen … liesel.vande…    rw     old
-## 5  Gebruik en onkosten dee… thierry.onke…    rw     old
-## 6  Uitgebreide lijst milie… floris.vande…     r     old
-## 7  Jarigentraktatie juli -… joris.vernai…    rw     old
-## 8  Buitenlandse zendingen …   toon.westra    rw     old
-## 9                 Begroting thierry.onke…    rw     old
-## 10 Uitgebreide lijst milie… floris.vande…    rw     old
-## # ... with 62 more rows, and 6 more variables: updated &lt;dttm&gt;,
+## 1  overzicht netwerken en … tessa.vansan…    rw     old
+## 2  LifeWatch budget en uit… peter.desmet…    rw     old
+## 3  Open biodiversity data …  peter.desmet    rw     old
+## 4  400+ Tools and innovati…    jeroensbox    rw     old
+## 5         Smartphone_uitrol      inbodata    rw     old
+## 6  DQ Tests-Assertions - P…      godfoder    rw     old
+## 7           IDC-budget 2017 daniel.duseu…    rw     old
+## 8  LifeWatch Monthly Activ…     info55896    rw     old
+## 9     Bird tracking devices  peter.desmet    rw     old
+## 10 2015.05.01 Contactgegev… thomas.timme…    rw     old
+## # ... with 157 more rows, and 6 more variables: updated &lt;dttm&gt;,
 ## #   sheet_key &lt;chr&gt;, ws_feed &lt;chr&gt;, alternate &lt;chr&gt;, self &lt;chr&gt;,
 ## #   alt_key &lt;chr&gt;</code></pre>
 </div>
@@ -311,6 +318,23 @@ <h2>I just want to load a google spreadsheet</h2>
 <pre class="r"><code>breedingbirds &lt;- redlists %&gt;% 
         gs_read(ws = &quot;BreedingBirds&quot;)</code></pre>
 <pre><code>## Accessing worksheet titled &#39;BreedingBirds&#39;.</code></pre>
+<pre><code>## 
+Downloading: 1.3 kB     
+Downloading: 1.3 kB     
+Downloading: 2.2 kB     
+Downloading: 2.2 kB     
+Downloading: 3 kB     
+Downloading: 3 kB     
+Downloading: 4.1 kB     
+Downloading: 4.1 kB     
+Downloading: 4.6 kB     
+Downloading: 4.6 kB     
+Downloading: 5.6 kB     
+Downloading: 5.6 kB     
+Downloading: 5.6 kB     
+Downloading: 5.6 kB     
+Downloading: 5.6 kB     
+Downloading: 5.6 kB</code></pre>
 <pre><code>## No encoding supplied: defaulting to UTF-8.</code></pre>
 <p>Inspecting the data of the sheet:</p>
 <pre class="r"><code>class(breedingbirds)</code></pre>
diff --git a/docs/data-handling-large-files-R.html b/docs/data-handling-large-files-R.html
diff --git a/docs/data-handling-large-files-R_files/figure-html/streamexample-1.png b/docs/data-handling-large-files-R_files/figure-html/streamexample-1.png
diff --git a/docs/development-styleguide-R.html b/docs/development-styleguide-R.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/development-styleguide-repos.html b/docs/development-styleguide-repos.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/index.html b/docs/index.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-R-admin.html b/docs/installation-R-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-R-user.html b/docs/installation-R-user.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-RStudio-admin.html b/docs/installation-RStudio-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-RStudio-user.html b/docs/installation-RStudio-user.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-git-admin.html b/docs/installation-git-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-latex-admin.html b/docs/installation-latex-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-pandoc-admin.html b/docs/installation-pandoc-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/docs/installation-rtools-admin.html b/docs/installation-rtools-admin.html
@@ -239,6 +239,12 @@
     <li>
       <a href="data-handling-googlesheet-R.html">Read data from GoogleSheet</a>
     </li>
+    <li>
+      <a href="data-handling-large-files-R.html">Reading and handling large data files in R</a>
+    </li>
+    <li>
+      <a href="data-handling-database-R.html">Read data from INBO databases (SQL Server) from R</a>
+    </li>
   </ul>
 </li>
       </ul>
diff --git a/img/access_dsn.png b/img/access_dsn.png
diff --git a/img/odbc_gegevensbron.png b/img/odbc_gegevensbron.png
diff --git a/source/_site.yml b/source/_site.yml
@@ -36,6 +36,8 @@ navbar:
         - text: "Read data from GoogleSheet"
           href: "data-handling-googlesheet-R.html"
         - text: "Reading and handling large data files in R"
-          href: "data-handling-large-files-R.html"          
+          href: "data-handling-large-files-R.html"
+        - text: "Read data from INBO databases (SQL Server) from R"
+          href: "data-handling-database-R.html"
 exclude: ["LICENSE", "Rprofile.site"]
 output_dir: "../docs"
diff --git a/source/data-handling-database-R.Rmd b/source/data-handling-database-R.Rmd
@@ -0,0 +1,152 @@
+---
+title: "Read data from INBO databases (SQL Server) with R(ODBC)"
+maintainer: "Stijn Van Hoey"
+date: "February 3, 2017"
+output: html_document
+---
+
+```{r setup, include=FALSE}
+knitr::opts_chunk$set(echo = TRUE)
+```
+
+To query data from a SQL database that is already accessible using MSAccess, such a database is also accessible from R using the package [RODBC](https://cran.r-project.org/web/packages/RODBC/index.html). This package enables the link in between R and the (remote) database. After installation of the package (`install.packages('RODBC')`), the package can be loaded:
+
+```{r, loadpackage, warning = FALSE, message = FALSE}
+library(RODBC)
+```
+
+For Windows users, the most important element is to know the so-called `DSN` (i.e. a registered Data Source Name). Actually, it is just the name of the database as it is known by your computer (and MS Access). The easiest way to check the `DSN` is to check the [*registered ODBC connections*](http://www.stata.com/support/faqs/data-management/configuring-odbc-win/) in the administrator tools menu. 
+
+For Dutch-speaking Windows 7 users: 
+
+    > Kies in het Configuratiescherm van Windows de optie Systeembeheer > Gegevensbronnen (ODBC). De optie Systeembeheer verschijnt in de categorie Systeem en onderhoud.
+
+You should see a list similar to the list underneath, with the names of the available DSN names enlisted:
+![odbc-connecties](../img/odbc_gegevensbron.png)
+
+An alternative way to check the DSN name of a database already working on with Access, is to check the DSN inside MS Access (in dutch, check menu item *Koppelingsbeheer*): 
+![access-dsn](../img/access_dsn.png)
+
+For example, the DSN name `UserFlora` or `Cydonia-prd` can be used to query these databases and extract data from it with similar queries to the one used in MSAccess. First of all, the connection with the database need to be established, by using the `odbcConnect` function, providing the DSN name as argument (Notice, you have to change the comments when running the tutorial in windows):
+
+```{r sqlconnection}
+# WINDOWS USERS (uncomment when working in Windows)
+#my_connection <- odbcConnect("UserFlora") # uncomment for windows users
+
+# LINUX/MAC USERS (comment out when working in Windows)
+my_connection <- odbcDriverConnect("Driver={ODBC Driver 13 for SQL Server};Server=inbosql04.inbo.be;Database=UserFlora;Trusted_Connection=yes;")
+```
+
+Once this connection is sucessfully established, the database can be queried. 
+
+**Remark for linux users:** When working in Linux, this setup requires an active *kerberos* session. More information about the setup and functionality will be provided in an upcoming tutorial.
+
+## Get a complete table from the databse
+
+The function `sqlFetch` can be used to load an entire table from a database. For example, to extract the `tblTaxon` table from the flora database:
+
+```{r querytable}
+rel_taxa <- sqlFetch(my_connection, "dbo.relTaxonTaxonGroep")
+head(rel_taxa)
+```
+
+with the connection `my_connection` made earlier is used as the first argument, the table name is the second argument.
+
+**Remark:** If you have no idea about the size of the table you're trying to load from the database, this could be rather tricky and cumbersome. Hence, it is probably better to only extract a portion of the table using a query.
+
+## Execute a query to the database
+
+The function `sqlQuery` provides more flexibilty as it can be used to try any SQL-query on the database. A complete introduction to the SQL language is out of scope here. We will focus on the application and the reusage of a query.
+
+```{r queryrun}
+meting <- sqlQuery(my_connection, paste("SELECT TOP 10 * FROM dbo.tblMeting", 
+                                        "WHERE Cor_X IS NOT NULL"))
+head(meting)
+```
+
+## Create and use query templates
+
+When you regularly use similar queries, with some minimal alterations, you do not want to copy/paste each time the entire query. It is prone to errors and you're script will become verbose. It is advisable to create query *templates*, that can be used within the `sqlQuery` function. 
+
+Consider the execution of the following query. We are interested in those records with valid X and Y coordinates for the measurement, based on a given dutch name:
+
+```{r queryexample}
+subset_meting <- sqlQuery(my_connection, 
+"SELECT meet.Cor_X
+	 , meet.Cor_Y
+	 , meet.MetingStatusCode
+	 , tax.NaamNederlands
+	 , tax.NaamWetenschappelijk
+	 , waar.IFBLHokID
+FROM  [dbo].[tblMeting] AS meet
+	LEFT JOIN [dbo].tblTaxon AS tax ON tax.ID = meet.TaxonID
+	LEFT JOIN [dbo].tblWaarneming AS waar ON waar.ID = meet.WaarnemingID
+WHERE meet.Cor_X IS NOT NULL
+	AND meet.Cor_X != 0
+	AND tax.NaamNederlands LIKE 'Wilde hyacint'")
+head(subset_meting)
+```
+
+If we need this query regularly, but each time using a different `tax.NaamNederlands` (only the name changes), it is worthwhile to invest some time in the creation of a small custom function that uses this query as a template. Let's create a function `flora_records_on_dutch_name` that takes a valid database connection and a given dutch name and returns the relevant subset of the data for this query:
+
+```{r querytemplate}
+
+flora_records_on_dutch_name <- function(dbase_connection, dutch_name) {
+     sqlQuery(dbase_connection, sprintf(
+            "SELECT meet.Cor_X
+            	 , meet.COr_Y
+            	 , meet.MetingStatusCode
+            	 , tax.NaamNederlands
+            	 , tax.NaamWetenschappelijk
+            	 , waar.IFBLHokID
+            FROM  [dbo].[tblMeting] meet
+            	LEFT JOIN [dbo].tblTaxon AS tax ON tax.ID = meet.TaxonID
+            	LEFT JOIN [dbo].tblWaarneming AS waar ON waar.ID = meet.WaarnemingID
+            WHERE meet.Cor_X IS NOT NULL
+            	AND meet.Cor_X != 0
+            	AND tax.NaamNederlands LIKE '%s'", dutch_name))
+}
+
+```
+
+Hence, instead of copy-pasting the whole query each time (which could be error-prone), we can reuse the function for different names:
+
+```{r}
+hyacint <- flora_records_on_dutch_name(my_connection, "Wilde hyacint")
+head(hyacint)
+```
+
+```{r}
+bosanemoon <- flora_records_on_dutch_name(my_connection, "Bosanemoon")
+head(bosanemoon)
+```
+
+**Remark:** Do not forget to close your connection when done finished. 
+
+```{r closeit}
+close(my_connection)
+```
+
+## The `sprintf` function {#sprintf}
+
+In order to accomplish the re-usage of a query for different input names (`dutch_name`), the `sprintf` function is used. The `sprintf` function provides the ability to combine text and variable values in a single charactor string (i.e. the query to execute). For each variable name required in the query (any part of your query you want to have interchangeable), a representation in the query is given by a `%` in combination with the data type you want to represent. The latter is an agreed format conversion, for example `%s` is a character string (e.g. the example of `dutch_name`) and `%i` is an integer value (e.g. `2`). More information about the different symbols is given in the documentation of `sprintf` (type `?sprintf` in the console). 
+
+```{r sprintfex1}
+a_name <- 'Jan'
+an_integer <- 3
+a_float <- 2.8
+sprintf('This prints a combination of a name: %s, an integer: %i and a float value: %f', a_name, an_integer, a_float)
+```
+
+For float values, you can further define the fixed number of decimals (`%f`) or a fixed number of significant digits with `%g`
+
+```{r sprintfex2}
+a_float <- 0.0008262372
+sprintf('This prints the same value with five: %.5f or two: %.2f decimal numbers, or a define a number of significant digits: %.5g or %.2g which is sometimes more useful.', a_float, a_float, a_float, a_float)
+```
+
+Similarly, this approach can be used to programmatically create template queries with adaptable input values. 
+
+
+
+
diff --git a/source/installation-pandoc-admin.Rmd b/source/installation-pandoc-admin.Rmd
@@ -5,7 +5,7 @@ maintainer: "Thierry Onkelinx"
 
 ## Windows
 
-Pandoc wordt automatisch geïnstalleerd als je RStudio installeerd.
+Pandoc wordt automatisch geïnstalleerd als je RStudio installeert.
 
 ## Ubuntu
 
