sphinx doc generation, and copy pages from wiki

Author: machow

docs/source/Home.md

@@ -0,0 +1,78 @@
+Home
+====
+
+pythonwhat?
+-----------
+
+A major part of DataCamp's interactive learning is centered around automated and meaningful feedback. When a student submits an incorrect answer, the system tells the student what he or she is doing wrong. This happens through so-called submission correctness tests, or SCTs. An SCT is a test script that compares the different steps in a student's submission to the ideal solution, and generates meaningful feedback along the way.
+
+`pythonwhat` is a Python package that can help you write these SCTs for interactive Python exercises on DataCamp. It allows you to easily compare parts in the student's submission with the solution code. `pythonwhat` provides a bunch of functions to test object definitions, function calls, function definitions, for loops, while loops, and many more. `pythonwhat` automatically generates meaningful feedback that's specific to the student's mistake; you can also choose to override this feedback with custom messages.
+
+Writing SCTs, for which `pythonwhat` is built, is only one part of creating DataCamp exercises. For general documentation on creating Python courses on DataCamp, visit the [Teach Documentation](https://www.datacamp.com/teach/documentation). To write SCTs for R exercises on DataCamp, have a look at [testwhat](https://github.com/datacamp/testwhat).
+
+How does it work?
+-----------------
+
+When a student starts an exercise on DataCamp, a Python session is started and the `pre_exercise_code` (PEC) is run. This code, that the author specifies, initializes the Python workspace with data, loads relevant packages etc, such that students can start coding the essence of the topics treated. Next, a separate solution process is created, in which the same PEC and actual solution code, also coded by the author, is executed.
+
+When a student submits an answer, his or her submission is executed and the output is shown in the IPython Shell. Then, the correctness of the submission is checked by executing the Submission Correctness Test, or SCT. Basically, your SCT is a Python script with calls to `pythonwhat` test functions. `pythonwhat` features a variety of functions to test a user's submission in different ways; examples are `test_object()`, `test_function()` and `test_output_contains()`. To do this properly, `pythonwhat` uses several resources:
+
+- The student submission as text, to e.g. figure out which functions have been called.
+- The solution code as text, to e.g. figure out whether the student called a particular function in the same way as it is called in the solution.
+- The student process, where the student code is executed, to e.g. figure out whether a certain object was created.
+- The solution process, where the solution code is executed, to e.g. figure out whether an object that the student created corresponds to the object that was created by the solution.
+- The output that's generated when executing the student code, to e.g. figure out if the student printed out something.
+
+If, during execution of the SCT, a test function notices a mistake, an appropriate feedback will be generated and presented to the student. It is always possible to override these feedback messages with your own messages. Defining custom feedback will make your SCTs longer and they may be error prone (typos, etc.), but they typically give the exercise a more natural and personalized feel.
+
+If all test functions pass, a success message is presented to the student. `pytonwhat` has some messages in store from which it can choose randomly, but you can override this with the `success_msg()` function.
+
+Overview
+--------
+
+To get started, make sure to check out the [Quickstart Guide](https://www.github.com/datacamp/pythonwhat/wiki/Quickstart_Guide).
+
+To robustly test the equality of objects, and results of evaluations, it has to fetch the information from the respective processes, i.e. the student and solution processes. By default, this is done through a process of 'dilling' and 'undilling', but it's also possible to define your own converters to customize the way objects and results are compared. For more background on this, check out the [Processes article](https://github.com/datacamp/pythonwhat/wiki/Processes). For some more background on the principle of 'sub-SCTs', i.e. sets of tests to be called on a particular part or a particular state of a student's submission, have a look at the [Sub-SCTs article](https://github.com/datacamp/pythonwhat/wiki/Sub-SCTs).
+
+The remainder of the wiki goes over every test function that `pythonwhat` features, explaining all arguments and covering different use cases. They will give you an idea of how, why and when to use them.
+
+**Basic functions**
+
+- Multiple choice exercises: [test_mc()](https://github.com/datacamp/pythonwhat/wiki/test_mc)
+- What student typed: [test_student_typed()](https://github.com/datacamp/pythonwhat/wiki/test_student_typed)
+- Which output is generated: [test_output_contains()](https://github.com/datacamp/pythonwhat/wiki/test_output_contains)
+- Value of object: [test_object()](https://github.com/datacamp/pythonwhat/wiki/test_object)
+- Usage of objects: [test_object_accessed()](https://github.com/datacamp/pythonwhat/wiki/tets_object_accessed)
+- Usage of function: [test_function()](https://github.com/datacamp/pythonwhat/wiki/test_function)
+- Usage of function, v2 (improved): [test_function_v2()](https://github.com/datacamp/pythonwhat/wiki/test_function_v2)
+- Usage of operators: [test_operator()](https://github.com/datacamp/pythonwhat/wiki/test_operator)
+- Package imports: [test_import()](https://github.com/datacamp/pythonwhat/wiki/test_import)
+
+**Logic-inducing functions**
+
+- If some tests fail, do other tests to pinpoint the problem: [test_correct()](https://www.github.com/datacamp/pythonwhat/wiki/test_correct)
+- Specify several tests, only one of which should pass: [test_or()](https://www.github.com/datacamp/pythonwhat/wiki/test_or)
+
+**Advanced functions**
+
+- Value of data frame: [test_data_frame()](https://github.com/datacamp/pythonwhat/wiki/test_data_frame)
+- Value of dictionary: [test_dictionary()](https://github.com/datacamp/pythonwhat/wiki/test_dictionary)
+- If ... else ... constructs: [test_if_else()](https://github.com/datacamp/pythonwhat/wiki/test_if_else)
+- For loops: [test_for_loop()](https://github.com/datacamp/pythonwhat/wiki/test_for_loop)
+- While loops: [test_while_loop()](https://github.com/datacamp/pythonwhat/wiki/test_while_loop)
+- Test value of object after expression: [test_object_after_expression()](https://github.com/datacamp/pythonwhat/wiki/test_object_after_expression)
+- Test output of an expression: [test_expression_output()](https://github.com/datacamp/pythonwhat/wiki/test_expression_output)
+- Test result of an expression: [test_expression_result()](https://github.com/datacamp/pythonwhat/wiki/test_expression_result)
+- Context managers, the with statement: [test_with()](https://github.com/datacamp/pythonwhat/wiki/test_with)
+- Test user-defined functions: [test_function_definition()](https://github.com/datacamp/pythonwhat/wiki/test_function_definition)
+- Test lambda function definitions: [test_lambda_function()](https://github.com/datacamp/pythonwhat/wiki/test_lambda_function)
+- Test comprehensions: [test_list_comp(), test_dict_comp() and test_generator_exp()](https://github.com/datacamp/pythonwhat/wiki/test_comprehension)
+- Test try except blcoks: [test_try_except()](https://github.com/datacamp/pythonwhat/test_try_except)
+
+All these functions are also documented inside the `pythonwhat` source code itself. The documentation there goes into the tiny details of all functions' implementations. Follow the steps in the README of this repository to generate a PDF version of the documentation.
+
+For more full examples of SCTs for Python exercises on DataCamp, check out the [source files of the introduction to Python course](http://www.github.com/datacamp/courses-intro-to-python). In the chapter files there, you can can see the SCTs that have been written for several exercises.
+
+To test your understanding of writing SCTs for Python exercises on the DataCamp platform, you can take the course [Writing SCTs with pythonwhat](https://www.datacamp.com/courses/writing-scts-with-pythonwhat) course.
+
+After reading through this documentation, we hope writing SCTs for Python exercises on DataCamp becomes a painless experience. If this is not the case and you think improvements to `pythonwhat` and this documentation are possible, [please let us know](mailto:content-engineering@datacamp.com)!

docs/source/Processes.md

@@ -0,0 +1,83 @@
+Processes
+=========
+
+As mentioned on the [wiki Homepage](https://github.com/datacamp/pythonwhat/wiki), DataCamp uses two separate processes. One process to run the solution code, and one process to run the student's submission. This way, `pythonwhat` has access to the 'ideal ending scenario' of an exercises; this makes it easier to write SCTs. Instead of having to specify which value an object should be, we can have `test_object()` look into the solution process and compare the object in that process with the object in the student process.
+
+### Problem
+
+Fetching Python objects or the results of running expressions inside a process is not straightforward. To be able to pull data from a process, Python needs to 'pickle' and 'unpickle' files: it converts the Python objects to a byte representation (pickling) that can be passed between processes, and then, inside the process that you want to work with the object, builds up the object from the byte representation again (unpickling).
+
+For the majority of Python objects, this conversion to and from a byte representation works fine, but for some objects, it doesn't. Even `dill`, and improved implementation of `pickle` that's being used in `pythonwhat`, doesn't flawlessly convert all Python objects out there.
+
+If you're writing an SCT with functions that require work in the solution process, such as `test_object()`, `test_function()`, and `test_function_definition()`, and then upload the exercise and test it on DataCamp, that you get backend errors that look like this:
+
+        ... dilling inside process failed - write manual converter
+        ... undilling of bytestream failed - write manual converter
+
+The first error tells you that 'dilling' - or 'pickling', converting the object to a bytestream representation, failed. The second error tells you that 'undilling' - or 'unpickling', converting the byte representation back to a Python object, failed. These errors will typically occur if you're dealing with exotic objects, such as objects that interface to files, connections to databases, etc.
+
+### Solution
+
+To be able to handle these errors, `pythonwhat` allows you to write your own converters for Python objects. Say, for example, that you're writing an exercise to import Excel data into Python, and you're using the `pandas` package. This is the solution and the corresponding SCT:
+
+        *** =solution
+        ```{python}
+        import pandas as pd
+        xl = pd.ExcelFile('battledeath.xlsx')
+        ```
+
+        *** =sct
+        ```{python}
+        test_object('xl')
+        ```
+
+Suppose now that objects such as `xl`, which are of the type `pandas.io.excel.ExcelFile`, can't be properly dilled and undilled. (Note: because of hardcoded converters inside `pythonwhat`, they can, see below). To make sure that you can still use `test_object('xl')` to test the equality of the `xl` object between student and solution process, you can manually define a converter with the `set_converter()` function. You can extend the SCT as follows:
+
+        *** =sct
+        ```
+        def my_converter(x):
+            return(x.sheet_names)
+        set_converter(key = "pandas.io.excel.ExcelFile", fundef = my_converter)
+        test_object('xl')
+        ```
+
+With a lambda function, it's even easier:
+
+        *** =sct
+        ```
+        set_converter(key = "pandas.io.excel.ExcelFile", fundef = lambda x: x.sheet_names)
+        test_object('xl')
+        ```
+
+The first arguemnt of `set_converter()`, the `key` takes the type of the object you want to add a manual converter for as a string. The second argument, `fundef`, takes a function definition, taking one argument and returning a single object. This function definition converts the exotic object into something more standard. In this case, the function converts the object of type `pandas.io.excel.ExcelFile` into a simple list of strings. A list of strings is something that can easily be converted into a bytestream and back into a Python object again, hence solving the problem.
+
+If you want to reuse the same manual converter over different exercises, you'll have to use `set_converter()` in every SCT.
+
+### Hardcoded converters
+
+Some converters will be required often. For example, the result of calling `.keys()` and `.items()` on dictionaries can't be dilled and undilled without extra work. To handle these common yet problematic situations, `pythonwhat` features a list of hardcoded converters. This list is [available in the source code](https://github.com/datacamp/pythonwhat/blob/master/pythonwhat/converters.py); feel free to do a pull request if you want to add more converts to this list. This will reduce the amount of code duplication you have to do if you want to reuse the same converter in different exercises.
+
+### Custom Equality
+
+The `set_converter()` function opens up possibilities for objects that can actually be dilled and undilled perfectly fine. Say you want to test a `numpy` array, but you only want to check only if the dimensions of the array the student codes up match those in the solution process. You can easily write a manual converter that overrides the typical dilling and undilling of Numpy arrays, implementing your custom equality behavior:
+
+        *** =solution
+        ```{python}
+        import numpy as np
+        my_array = np.array([[1,2], [3,4], [5,6]])
+        ```
+
+        *** =sct
+        ```
+        set_converter(key = "numpy.ndarray", fundef = lambda x: x.shape)
+        test_object('my_array')
+        ```
+
+Both of the following submissions will be accepted by this SCT:
+
+- `my_array = np.array([[1,2], [3,4], [5,6]])`
+- `my_array = np.array([[0,0], [0,0], [5,6]])`
+
+
+
+

docs/source/Sub-SCTs.md

@@ -0,0 +1,133 @@
+Composite SCTs
+==============
+
+Several functions, such as `test_correct()`, `test_or()`, `test_function_definition()`, among others, also have arguments that expect another set of tests. This article will explain the different ways of specifying these 'sub-tests'.
+
+Let's take the example of `test_correct()`; this function takes two sets of tests. The first set is executed, and if it's fine, the second set is left alone. If the first set of tests results in an error, the second set is executed and the feedback is logged.
+
+### Example 1
+
+As an example, suppose you want the student to calculate the mean of a Numpy array `arr` and store it in `res`. A possible solution could be:
+
+    *** =solution
+    ```{python}
+    # Import numpy and create array
+    import numpy as np
+    arr = np.array([1, 2, 3, 4, 5, 6])
+
+    # Calculate result
+    result = np.mean(arr)
+    ```
+
+The first part of the tests here would be to check `result`. If `result` is not correct, you want to check whether `np.mean()` has been called.
+
+The most concise way to do this, is with lambda functions; you specify two sets of tests, that in this case consist of one test each:
+
+    *** =sct
+    ```{python}
+    test_correct(lambda: test_object('result'),
+                 lambda: test_function('numpy.mean'))
+    success_msg("You own numpy!")
+    ```
+
+Another way to do this, is by specifying two separate function definitions, e.g. `check` and `diagnose`, and pass these function definitions to `test_correct()`, as shown below. Notice that you have to pass the actual function definition, so `check` and `diagnose` without parentheses:
+
+    *** =sct
+    ```{python}
+    def check():
+        test_object('result')
+
+    def diagnose():
+        test_function('numpy.mean')
+
+    test_correct(check, diagnose)
+    success_msg("You own numpy!")
+    ```
+
+Of course, you can also use a lambda function for one test set, and a function definition for the other:
+
+    *** =sct
+    ```{python}
+    def check():
+        test_object('result')
+
+    test_correct(check,
+                 lambda: test_function('numpy.mean'))
+    success_msg("You own numpy!")
+    ```
+
+
+### Example 2
+
+When writing SCTs for more complicated exercises, you'll probably want to pass along several tests to an argument.
+
+Suppose that you expect the student to create an object `result` once more, but this time it's the sum of calling the `np.mean()` function twice; once on `arr1`, once on `arr2`:
+
+    *** =solution
+    ```{python}
+    # Import numpy and create array
+    import numpy as np
+    arr1 = np.array([1, 2, 3, 4, 5, 6])
+    arr2 = np.array([6, 5, 4, 3, 2, 1])
+
+    # Calculate result
+    result = np.mean(arr) + np.mean(arr)
+    ```
+
+Now, in the 'digging deeper' part of `test_correct()`, you want to check the `np.mean()` function twice. To do this, you'll want to use a function definition; lambda functions are not practical anymore:
+
+    *** =sct
+    ```{python}
+    def diagnose():
+        test_function('numpy.mean', index = 1)
+        test_function('numpy.mean', index = 2)
+
+    test_correct(lambda: test_object('result'),
+                 diagnose)
+    success_msg("You own numpy!")
+    ```
+
+### Example 3: custom feedback
+
+To pass custom feedback feedback messages to these sub-SCTs, you should define the custom message inside the functions themselves, inside the function definition, or define it beforehand and pass it along as a default argument. Let's revisit the first example:
+
+    *** =solution
+    ```{python}
+    # Import numpy and create array
+    import numpy as np
+    arr = np.array([1, 2, 3, 4, 5, 6])
+
+    # Calculate result
+    result = np.mean(arr)
+    ```
+
+Here are the three different SCT options, this time with custom feedback:
+
+    *** =sct
+    ```{python}
+    # OPTION 1
+    msg1 = "Are you sure `result` is correct?"
+    msg2 = "Your call of `np.mean()` isn't right."
+    test_correct(lambda msg=msg1: test_object('result', incorrect_msg = msg),
+                 lambda msg=msg2: test_function('numpy.mean', incorrect_msg = msg))
+    success_msg("You own numpy!")
+
+    # OPTION 2
+    def check():
+        test_object('result', incorrect_msg = "Are you sure `result` is correct?")
+
+    def diagnose():
+        test_function('numpy.mean', incorrect_msg = "Your call of `np.mean()` isn't right.")
+
+    test_correct(check, diagnose)
+    success_msg("You own numpy!")
+
+
+    # OPTION 3
+    def check():
+        test_object('result', incorrect_msg = "Are you sure `result` is correct?")
+
+    test_correct(check,
+                 lambda: test_function('numpy.mean', incorrect_msg = "Your call of `np.mean()` isn't right."))
+    success_msg("You own numpy!")
+    ```