Skip to content

Commit 7c95e4a

Browse files
AleMoralesAleMorales
authored
Update use&dev_packages.md
Addresses issue #11
1 parent 5d14355 commit 7c95e4a

1 file changed

Lines changed: 147 additions & 146 deletions

File tree

‎docs/src/developers/use&dev_packages.md‎

Lines changed: 147 additions & 146 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Package and Environment Management for VPL
22

3-
Ana Ernst & Alejandro Morales
4-
Centre for Crop Systems Analysis - Wageningen University
3+
Ana Ernst & Alejandro Morales
4+
Centre for Crop Systems Analysis - Wageningen University
55

66
This guide helps VPL users interested in (1) managing packages within the VPL-verse, (2) managing reproducible environments, and, (3) developing models into VPL-modules, customization, and extending VPL's functionality under their authorship. It equips them with essential Julia tools and functionalities to take control of these aspects, enhancing the VPL experience.
77

@@ -11,15 +11,15 @@ The guide introduces the user to the Julia programming language's package manage
1111

1212
Unlike traditional package managers, that install and manage a single global set of packages, [Pkg.jl](https://pkgdocs.julialang.org/v1/) does things differently, it uses **environments** - independent sets of packages that can be local to an individual project, and can be shared and selected by name.
1313

14-
In each of these environments, there is a neat list of the packages associated with the project and their exact versions, kept in a special file called `Manifest.toml`. You can check this file in the project repository and keep track of the versions of the different packages that you use on your project, significantly improving the reproducibility of projects.
14+
In each of these environments, there is a neat list of the packages associated with the project and their exact versions, kept in a special file called `Manifest.toml`. You can check this file in the project repository and keep track of the versions of the different packages that you use on your project, significantly improving the reproducibility of projects.
1515

16-
So, if you ever want to work on a project from a different computer or share with with someone else, you can just bring back the environment using its manifest file. That way, you're all set with the right packages without any fuss.
16+
So, if you ever want to work on a project from a different computer or share with with someone else, you can just bring back the environment using its manifest file. That way, you're all set with the right packages without any fuss.
1717

1818
Julia environments are *stackable* - you can overlay one environment with another and have access to additional packages outside of the original environment.
1919

20-
This makes it easy to work on a project, which provides the primary environment, while still having access from the REPL to all your usual dev tools like profilers, debuggers, and so on, just by having an environment including these dev tools later in the load path.
20+
This makes it easy to work on a project, which provides the primary environment, while still having access from the REPL to all your usual dev tools like profilers, debuggers, and so on, just by having an environment including these dev tools later in the load path.
2121

22-
So, it's like having your main work area and then, when needed, you can simply add more tools on top of it without disturbing your primary setup. This keeps things organized and makes it easy to switch between different tasks in Julia.
22+
So, it's like having your main work area and then, when needed, you can simply add more tools on top of it without disturbing your primary setup. This keeps things organized and makes it easy to switch between different tasks in Julia.
2323

2424
## Getting started with Pkg.jl and environments
2525

@@ -37,46 +37,46 @@ To better understand how Julia works with environments, we have two different ty
3737

3838
* **Project Environment**
3939

40-
It is a directory with a *project file* and a *manifest file*. A *project file* determines what are the names and identities of the direct dependencies of a project. A *manifest file* gives a complete dependency graph, including all direct and indirect dependencies, exact versions of each dependency, and sufficient information to locate and load the correct version.
40+
It is a directory with a *project file* and a *manifest file*. A *project file* determines what are the names and identities of the direct dependencies of a project. A *manifest file* gives a complete dependency graph, including all direct and indirect dependencies, exact versions of each dependency, and sufficient information to locate and load the correct version.
4141

42-
These types of environments provide reproducibility. By storing a project environment in version control, such as a Git repository, along with the rest of project's source code, you can reproduce the exact state of the project and all of its dependencies. The manifest file, in particular, captures the exact version of every dependency, which makes it possible for Pkg to retrieve the correct versions and be sure that you are running the exact code that was recorded for all dependencies. These are fit to run your simulations, and share such files with others.
42+
These types of environments provide reproducibility. By storing a project environment in version control, such as a Git repository, along with the rest of project's source code, you can reproduce the exact state of the project and all of its dependencies. The manifest file, in particular, captures the exact version of every dependency, which makes it possible for Pkg to retrieve the correct versions and be sure that you are running the exact code that was recorded for all dependencies. These are fit to run your simulations, and share such files with others.
4343

4444
* **Package Directory**
4545

46-
It is directory that contains a tree of subdirectories, and forms an implicit environment. Each subdirectory contains a specific package - if the `X` is is a subdirectory of a package directory and `X/src/X.jl` exists, we can access to the package `X` from the package directory.
46+
It is directory that contains a tree of subdirectories, and forms an implicit environment. Each subdirectory contains a specific package - if the `X` is is a subdirectory of a package directory and `X/src/X.jl` exists, we can access to the package `X` from the package directory.
4747

48-
Package directories are useful when you want to put a set of tools (in the format of packages) somewhere and be able to directly use them, without needing to create a project environment for them.
48+
Package directories are useful when you want to put a set of tools (in the format of packages) somewhere and be able to directly use them, without needing to create a project environment for them.
4949

5050
You can overlay these types of environments and obtained a **stacked environment** - an ordered set of project environments and package directories that make a single composite environment. The precedent and visibility rules combined determine which packages are available and where they get loaded from.
5151

5252
## Creating environments and using someone else's project
5353

5454
You may have noticed the `(@v1.8)` in the REPL prompt. So far we have added packages to the default environment at `~/.julia/environments/v1.8`. However, it is easy to create independent projects. This let us know that `v1.8` is the **active environment** - this is the environment that will be modified by `Pkg` commands above.
5555

56-
To set the active environment, you can use `activate`. In case this, we want to create a new active environment. `Pkg` will let us know that we are creating a new environment that will be stored in your `~/directory`.
56+
To set the active environment, you can use `activate`. In case this, we want to create a new active environment. `Pkg` will let us know that we are creating a new environment that will be stored in your `~/directory`.
5757

58-
````julia
59-
(@v1.8) pkg> activate MyProject
60-
Activating new environment at `~/MyProject/Project.toml`
58+
````julia
59+
(@v1.8) pkg> activate MyProject
60+
Activating new environment at `~/MyProject/Project.toml`
6161

62-
(MyProject) pkg> st
63-
Status `~/MyProject/Project.toml` (empty project)
64-
````
62+
(MyProject) pkg> st
63+
Status `~/MyProject/Project.toml` (empty project)
64+
````
6565

6666
Note that the REPL prompt changes when the new project is activated. Until a package is added, there are no files in this environment and the directory of this environment might not be created.
6767

6868
When you plan on using someone else's project, you can simply clone their project with `git clone`.
6969

70-
````julia
71-
shell> git clone https://github.com/JuliaLang/Example.jl.git
72-
Cloning into 'Example.jl'...
73-
...
74-
(@v1.8) pkg> activate Example.jl
75-
Activating project at `~/Example.jl`
76-
(Example) pkg> instantiate
77-
No Changes to `~/Example.jl/Project.toml`
78-
No Changes to `~/Example.jl/Manifest.toml`
79-
````
70+
````julia
71+
shell> git clone https://github.com/JuliaLang/Example.jl.git
72+
Cloning into 'Example.jl'...
73+
...
74+
(@v1.8) pkg> activate Example.jl
75+
Activating project at `~/Example.jl`
76+
(Example) pkg> instantiate
77+
No Changes to `~/Example.jl/Project.toml`
78+
No Changes to `~/Example.jl/Manifest.toml`
79+
````
8080

8181
If the project contains a `Manifest.toml` file, this will install the packages in the same state that is given by that manifest. If not, by default the latest versions of the dependencies compatible with the project will be installed.
8282

@@ -92,22 +92,23 @@ The control **activate** by itself does not install missing dependencies. Howeve
9292

9393
The [PkgTemplates.jl](https://github.com/JuliaCI/PkgTemplates.jl) package offers an easy, repeatable, and customizable way to generate files for a new package. Likewise, usage is pretty straightforward. The following example includes the plugins suitable for a project hosted on GitHub, with some other customizations:
9494

95-
````julia
96-
using PkgTemplates
97-
t = Template(;
98-
user="my-username",
99-
dir="~/code",
100-
authors="author_name",
101-
julia=v"1.1",
102-
plugins=[
103-
License(; name="MPL"),
104-
Git(; manifest=true, ssh=true),
105-
GitHubActions(; x86=true),
106-
Codecov(),
107-
Documenter{GitHubActions}(),
108-
Develop()])
109-
t("MyPkg")
110-
````
95+
````julia
96+
using PkgTemplates
97+
t = Template(;
98+
user="my-username",
99+
dir="~/code",
100+
authors="author_name",
101+
julia=v"1.1",
102+
plugins=[
103+
License(; name="MPL"),
104+
Git(; manifest=true, ssh=true),
105+
GitHubActions(; x86=true),
106+
Codecov(),
107+
Documenter{GitHubActions}(),
108+
Develop()])
109+
t("MyPkg")
110+
````
111+
111112
Keyword arguments for `PkgTemplates.Template` object type:
112113

113114
* *User options:*
@@ -121,31 +122,31 @@ Keyword arguments for `PkgTemplates.Template` object type:
121122

122123
This creates a new package called `MyPkg`, with a new project `MyPkg.jl` in the subdirectory `src`. You can visualize directory structure with the `tree` command in the Julia REPL.
123124

124-
````julia
125-
shell> tree MyPkg/
126-
MyPkg/
127-
├── Project.toml
128-
├── Manifest.toml
129-
└── src
130-
└── MyPkg.jl
131-
2 directories, 3 files
132-
````
125+
````julia
126+
shell> tree MyPkg/
127+
MyPkg/
128+
├── Project.toml
129+
├── Manifest.toml
130+
└── src
131+
└── MyPkg.jl
132+
2 directories, 3 files
133+
````
133134

134135
The content of `src/MyPkg.jl` can be modified:
135136

136-
````julia
137-
module MyPkg
138-
#Add functions to the module
139-
greet() = print(“Hello world!”)
140-
end
137+
````julia
138+
module MyPkg
139+
#Add functions to the module
140+
greet() = print(“Hello world!”)
141+
end
141142

142-
pkg> activate ./MyPkg #Activate the project by using the path to the directory where it is installed
143-
#Load the package
144-
julia> import MyPkg
145-
julia> MyPkg.greet()
143+
pkg> activate ./MyPkg #Activate the project by using the path to the directory where it is installed
144+
#Load the package
145+
julia> import MyPkg
146+
julia> MyPkg.greet()
146147

147-
Hello world!
148-
````
148+
Hello world!
149+
````
149150

150151
## Styling guidelines
151152

@@ -155,63 +156,63 @@ Package and function names should be sensible to most Julia users, even to those
155156

156157
Back to `Pkg.jl`; let’s say we want to use a standard library package (e.g., `Random`) and a registered package (e.g. `JSON`). We simply add these packages.
157158

158-
````julia
159-
(MyPkg) pkg> add Random JSON
160-
Resolving package versions...
161-
Updating `~/MyPkg/Project.toml`
162-
[682c06a0] + JSON v0.21.3
163-
[9a3f8284] + Random
164-
Updating `~/MyPkg/Manifest.toml`
165-
[682c06a0] + JSON v0.21.3
166-
[69de0a69] + Parsers v2.4.0
167-
[ade2ca70] + Dates
168-
...
169-
````
159+
````julia
160+
(MyPkg) pkg> add Random JSON
161+
Resolving package versions...
162+
Updating `~/MyPkg/Project.toml`
163+
[682c06a0] + JSON v0.21.3
164+
[9a3f8284] + Random
165+
Updating `~/MyPkg/Manifest.toml`
166+
[682c06a0] + JSON v0.21.3
167+
[69de0a69] + Parsers v2.4.0
168+
[ade2ca70] + Dates
169+
...
170+
````
170171

171172
Both `Random` and `JSON` now got added to the project’s `Project.toml` file, and the resulting dependencies to the `Manifest.toml` file. We can now both use Random and JSON in our package, by changing `src/MyPkg.jl` again.
172173

173-
````julia
174-
module MyPkg
175-
#Importing packages
176-
import Random
177-
import JSON
178-
#Add functions to the module
179-
greet() = print("Hello World!")
180-
greet_alien() = print("Hello ", Random.randstring(8))
181-
end
182-
````
174+
````julia
175+
module MyPkg
176+
#Importing packages
177+
import Random
178+
import JSON
179+
#Add functions to the module
180+
greet() = print("Hello World!")
181+
greet_alien() = print("Hello ", Random.randstring(8))
182+
end
183+
````
183184

184185
To reload a package in Julia without restarting the Julia session, you can use the [`Revise.jl`](https://timholy.github.io/Revise.jl/stable/) package whenever you want to work interactively and see immediate updates to your code.
185186

186-
````julia
187-
using Revise #Important to load Revise.jl before your package
188-
using MyPkg
189-
#Now you can call the updated greet_alien() function
190-
julia> MyPkg.greet_alien()
187+
````julia
188+
using Revise #Important to load Revise.jl before your package
189+
using MyPkg
190+
#Now you can call the updated greet_alien() function
191+
julia> MyPkg.greet_alien()
191192

192-
Hello aT157rHV
193-
````
193+
Hello aT157rHV
194+
````
194195

195196
## Adding tests
196197

197198
To add tests to a package, we have to (1) create a directory for tests, (2) create and write the script file, (3) execute tests.
198199

199-
````julia
200-
#Create path to test directory
201-
mkpath("MyPkg/test")
202-
#Create simple test script file in MyPkg/test/runtests.jl
203-
write("MyPkg/test/runtests.jl",
204-
"""
205-
#You can add specific tests here or edit seperatly the file itself
206-
println("Testing...")
207-
""");
208-
#This command will run the tests specified in MyPkg/test/runtests.jl
209-
(MyPkg) pkg> test
210-
Testing MyPkg
211-
Resolving package versions...
212-
Testing...
213-
Testing MyPkg tests passed
214-
````
200+
````julia
201+
#Create path to test directory
202+
mkpath("MyPkg/test")
203+
#Create simple test script file in MyPkg/test/runtests.jl
204+
write("MyPkg/test/runtests.jl",
205+
"""
206+
#You can add specific tests here or edit seperatly the file itself
207+
println("Testing...")
208+
""");
209+
#This command will run the tests specified in MyPkg/test/runtests.jl
210+
(MyPkg) pkg> test
211+
Testing MyPkg
212+
Resolving package versions...
213+
Testing...
214+
Testing MyPkg tests passed
215+
````
215216

216217
## Advanced considerations
217218

@@ -229,20 +230,20 @@ Before formally registering your package, there are some more advanced nuanced i
229230

230231
Use [`ArgCheck.jl`](https://github.com/jw3126/ArgCheck.jl) to perform checks on function arguments for public API (one may also use dedicated types to encapsulate these checks). VPL should have a strong preference for defensive programming even if this comes at the expense of runtime performance (but do try to avoid performance hits and repeating assertions).
231232

232-
````julia
233-
#After installing ArgCheck - Pkg.add("ArgCheck")
234-
using ArgCheck
235-
#Example in function
236-
function f(x,y)
237-
@argcheck x < y
238-
#The rest of the function
239-
end
240-
241-
f(0,0)
242-
ERROR: ArgumentError: x < y must hold. Got
243-
x => 0
244-
y => 0
245-
````
233+
````julia
234+
#After installing ArgCheck - Pkg.add("ArgCheck")
235+
using ArgCheck
236+
#Example in function
237+
function f(x,y)
238+
@argcheck x < y
239+
#The rest of the function
240+
end
241+
242+
f(0,0)
243+
ERROR: ArgumentError: x < y must hold. Got
244+
x => 0
245+
y => 0
246+
````
246247

247248
* Try to achieve type stability, by using `@infered` in tests and statistically-size arrays, as often as possible.
248249
* Avoid containers with abstract types.
@@ -267,29 +268,29 @@ Use [`ArgCheck.jl`](https://github.com/jw3126/ArgCheck.jl) to perform checks on
267268
All types created for the tests should be included in a module and all tests should be run inside a (let … end) block.
268269
* Test should cover 100% of source code (at least locally). The following is a template that can be used (normally located in a local dev folder not synced with Github):
269270

270-
````julia
271-
# Run tests and collect statistics on code coverage
272-
import Pkg
273-
Pkg.test("PlantRayTracer", coverage = true)
274-
275-
# Process the coverage data using Coverage.jl
276-
using Coverage
277-
# process '*.cov' files
278-
coverage = process_folder("src") # defaults to src/; alternatively, supply the folder name as argument
279-
# process '*.info' files, if you collected them
280-
coverage = merge_coverage_counts(coverage,
281-
filter!(let prefixes = (joinpath(pwd(), "src", ""))
282-
c -> any(p -> startswith(c.filename, p), prefixes)
283-
end,
284-
LCOV.readfolder("test")))
285-
# Get total coverage for all Julia files
286-
covered_lines, total_lines = get_summary(coverage)
287-
covered_lines / total_lines * 100 # 87%
288-
289-
# Clean up all the coverage files
290-
clean_folder("src")
291-
clean_folder("test")
292-
````
271+
````julia
272+
# Run tests and collect statistics on code coverage
273+
import Pkg
274+
Pkg.test("PlantRayTracer", coverage = true)
275+
276+
# Process the coverage data using Coverage.jl
277+
using Coverage
278+
# process '*.cov' files
279+
coverage = process_folder("src") # defaults to src/; alternatively, supply the folder name as argument
280+
# process '*.info' files, if you collected them
281+
coverage = merge_coverage_counts(coverage,
282+
filter!(let prefixes = (joinpath(pwd(), "src", ""))
283+
c -> any(p -> startswith(c.filename, p), prefixes)
284+
end,
285+
LCOV.readfolder("test")))
286+
# Get total coverage for all Julia files
287+
covered_lines, total_lines = get_summary(coverage)
288+
covered_lines / total_lines * 100 # 87%
289+
290+
# Clean up all the coverage files
291+
clean_folder("src")
292+
clean_folder("test")
293+
````
293294

294295
**Updating an existing package**
295296
---
@@ -300,7 +301,7 @@ All changes to a package should occur in a branch and a pull request should be u
300301
* Significant features have been added
301302

302303
Check that all the tests are successful in a dedicated module. This requires running `Pkg.test(<package>)` rather than running the `runtest.jl` file in Julia. This will tell you if there are any dependencies missing in the Project.toml of the tests.
303-
You may use the development version of a dependency while testing locally, but CI will fail until the changes in the dependency are registered.
304+
You may use the development version of a dependency while testing locally, but CI will fail until the changes in the dependency are registered.
304305

305306
*Increasing version number:*
306307

0 commit comments

Comments
 (0)