You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _sources/config_examples.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,6 +46,7 @@ Our internal configurations for each pipeline are as follows:
46
46
| EBS_VOL_SIZE (if using S3 mounted as a file system) | 22 | 22 | 22 | 22 | 22 | Files are read directly off of S3, mounted as a file system when `DOWNLOAD_FILES = False`. |
47
47
| EBS_VOL_SIZE (if downloading files) | 22 | 200 | 22 | 22 | 40 | Files are downloaded to the EBS volume when `DOWNLOAD_FILES = True`. |
Copy file name to clipboardExpand all lines: _sources/costs.md
+18-6Lines changed: 18 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,18 +2,28 @@
2
2
3
3
Distributed-CellProfiler is run by a series of three commands, only one of which incurs costs at typical scale of usage:
4
4
5
-
[`setup`](step_1_configuration.md) creates a queue in SQS and a cluster, service, and task definition in ECS.
6
-
ECS is entirely free.
5
+
[`setup`](step_1_configuration.md) creates a queue in SQS and a cluster, service, and task definition in ECS.
6
+
ECS is entirely free.
7
7
SQS queues are free to create and use up to 1 million requests/month.
8
8
9
9
[`submitJobs`](step_2_submit_jobs.md) places messages in the SQS queue which is free (under 1 million requests/month).
10
10
11
-
[`startCluster`](step_3_start_cluster.md) is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard.
11
+
[`startCluster`](step_3_start_cluster.md) is the only command that incurs costs with initiation of your spot fleet request, creating machine alarms, and optionally creating a run dashboard.
12
12
13
-
The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use.
13
+
The spot fleet is the major cost of running Distributed-CellProfiler, exact pricing of which depends on the number of machines, type of machines, and duration of use.
14
14
Your bid is configured in the [config file](step_1_configuration.md).
15
+
Simple spot fleet configurations can be minimized by:
16
+
17
+
1) Optimize `MACHINE_TYPE` and `EBS_VOL_SIZE` based on the actual memory and harddrive needs of your run.
18
+
2) When possible, mount your S3 bucket using S3FS so that you can set `DOWNLOAD_FILES = 'False'` to not incur file egress costs.
19
+
See [Step 1 Configuration](step_1_configuration.md) for more information.
20
+
Data egress charges range for various reasons including traversing AWS regions and/or AWS availability zones but are [often $0.08–$0.12 per GB](https://aws.amazon.com/blogs/apn/aws-data-transfer-charges-for-server-and-serverless-architectures/).
21
+
3) Set `ASSIGN_IP = 'False'` so that you don't pay for IPv4 addresses per EC2 instance in your spot fleet.
22
+
Public IPv4 costs are minimal ([$0.005/IP/hour as of February 1, 2024](https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/)) but there is no need to incur even this minimal cost unless you have a specific need for it.
23
+
See [Step 1 Configuration](step_1_configuration.md) for more information.
15
24
16
25
Spot fleet costs can be minimized/stopped in multiple ways:
26
+
17
27
1) We encourage the use of [`monitor`](step_4_monitor.md) during your job to help minimize the spot fleet cost as it automatically scales down your spot fleet request as your job queue empties and cancels your spot fleet request when you have no more jobs in the queue.
18
28
Note that you can also perform a more aggressive downscaling of your fleet by monitor by engaging Cheapest mode (see [`more information here`](step_4_monitor.md)).
19
29
2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling).
@@ -23,14 +33,16 @@ Note that you can also perform a more aggressive downscaling of your fleet by mo
23
33
After the spot fleet has started, a Cloudwatch instance alarm is automatically placed on each instance in the fleet.
24
34
Cloudwatch instance alarms [are currently $0.10/alarm/month](https://aws.amazon.com/cloudwatch/pricing/).
25
35
Cloudwatch instance alarm costs can be minimized/stopped in multiple ways:
36
+
26
37
1) If you run monitor during your job, it will automatically delete Cloudwatch alarms for any instance that is no longer in use once an hour while running and at the end of a run.
27
38
2) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to delete Cloudwatch alarms for any instance that is no longer in use.
28
39
3) In [AWS Cloudwatch console](https://console.aws.amazon.com/cloudwatch/) you can select unused alarms by going to Alarms => All alarms. Change Any State to Insufficient Data, select all alarms, and then Actions => Delete.
29
40
4) We provide a [hygiene script](hygiene.md) that will clean up old alarms for you.
30
41
31
-
Cloudwatch Dashboards [are currently free](https://aws.amazon.com/cloudwatch/pricing/) for 3 Dashboards with up to 50 metrics per month and are $3 per dashboard per month after that.
42
+
Cloudwatch Dashboards [are currently free](https://aws.amazon.com/cloudwatch/pricing/) for 3 Dashboards with up to 50 metrics per month and are $3 per dashboard per month after that.
32
43
Cloudwatch Dashboard costs can be minimized/prevented in multiple ways:
44
+
33
45
1) You can choose not to have Distributed-CellProfiler create a Dashboard by setting `CREATE_DASHBOARD = 'False'` in your [config file](step_1_configuration.md).
34
46
2) We encourage the use of [`monitor`](step_4_monitor.md) during your job as if you have set `CLEAN_DASHBOARD = 'True'` in your [config file](step_1_configuration.md) it will automatically delete your Dashboard when your job is done.
35
47
3) If your job is finished, you can still initiate [`monitor`](step_4_monitor.md) to perform the same cleanup (without the automatic scaling).
36
-
4) You can manually delete Dashboards in the [Cloudwatch Console]((https://console.aws.amazon.com/cloudwatch/)) by going to Dashboards, selecting your Dashboard, and selecting Delete.
48
+
4) You can manually delete Dashboards in the [Cloudwatch Console]((https://console.aws.amazon.com/cloudwatch/)) by going to Dashboards, selecting your Dashboard, and selecting Delete.
Copy file name to clipboardExpand all lines: _sources/step_1_configuration.md
+18-6Lines changed: 18 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,6 +17,7 @@ It need not be unique, but it should be descriptive enough that you can tell job
17
17
***
18
18
19
19
### AWS GENERAL SETTINGS
20
+
20
21
These are settings that will allow your instances to be configured correctly and access the resources they need- see [Step 0: Prep](step_0_prep.md) for more information.
21
22
22
23
Bucket configurations allow you to read/write from/to different bucket in different accounts from where you are running DCP.
@@ -48,15 +49,23 @@ Distinct clusters for each job are not necessary, but if you're running multiple
48
49
***MACHINE_PRICE:** How much you're willing to pay per hour for each machine launched.
49
50
AWS has a handy [price history tracker](https://console.aws.amazon.com/ec2sp/v1/spot/home) you can use to make a reasonable estimate of how much to bid.
50
51
If your jobs complete quickly and/or you don't need the data immediately you can reduce your bid accordingly; jobs that may take many hours to finish or that you need results from immediately may justify a higher bid.
52
+
See also [AWS on-demand pricing](https://aws.amazon.com/ec2/pricing/on-demand/) to compare the cost savings of using spot fleets.
51
53
***EBS_VOL_SIZE:** The size of the temporary hard drive associated with each EC2 instance in GB.
52
54
The minimum allowed is 22.
53
55
If you have multiple Dockers running per machine, each Docker will have access to (EBS_VOL_SIZE/TASKS_PER_MACHINE)- 2 GB of space.
54
56
***DOWNLOAD_FILES:** Whether or not to download the image files to the EBS volume before processing, as opposed to accessing them all from S3FS.
55
57
This typically requires a larger EBS volume (depending on the size of your image sets, and how many sets are processed per group), but avoids occasional issues with S3FS that can crop up on longer runs.
58
+
By default, DCP uses S3FS to mount the S3 `SOURCE_BUCKET` as a pseudo-file system on each EC2 instance in your spot fleet to avoid file download.
59
+
If you are unable to mount the `SOURCE_BUCKET` (perhaps because of a permissions issue) you should proceed with `DOWNLOAD_FILES = 'True'`.
60
+
***ASSIGN_IP:** Whether or not to assign an a public IPv4 address to each instance in the spot fleet.
61
+
If set to 'False' will overwrite whatever is in the Fleet file.
62
+
If set to 'True' will respect whatever is in the Fleet file.
63
+
Distributed-CellProfiler originally defaulted to assign an IP address to each instance so that one could connect to the instance for troubleshooting but that need has been mostly obviated by the level of logging currently in DCP.
56
64
57
65
***
58
66
59
67
### DOCKER INSTANCE RUNNING ENVIRONMENT
68
+
60
69
***DOCKER_CORES:** How many copies of your script to run in each Docker container.
61
70
***CPU_SHARES:** How many CPUs each Docker container may have.
62
71
***MEMORY:** How much memory each Docker container may have.
@@ -83,8 +92,9 @@ See [Step 0: Prep](step_0_prep.med) for more information.
83
92
84
93
***
85
94
86
-
### MONITORING
87
-
***AUTO_MONITOR:** Whether or not to have Auto-Monitor automatically monitor your jobs.
95
+
### MONITORING
96
+
97
+
***AUTO_MONITOR:** Whether or not to have Auto-Monitor automatically monitor your jobs.
88
98
89
99
***
90
100
@@ -111,6 +121,7 @@ Useful when trying to detect jobs that may have exported smaller corrupted files
111
121
***
112
122
113
123
### CELLPROFILER SETTINGS
124
+
114
125
***ALWAYS CONTINUE:** Whether or not to run CellProfiler with the --always-continue flag, which will keep CellProfiler from crashing if it errors.
115
126
Use with caution.
116
127
This can be particularly helpful in jobs where a large number of files are loaded in a single run (such as during illumination correction) so that a corrupted or missing file doesn't prevent the whole job completing.
@@ -120,6 +131,7 @@ We suggest using this setting in conjunction with a small number of JOB_RETRIES.
120
131
***
121
132
122
133
### PLUGINS
134
+
123
135
***USE_PLUGINS:** Whether or not you will be using external plugins from the CellProfiler-plugins repository.
124
136
When True, passes the `--plugins-directory` flag to CellProfiler.
125
137
Defaults to the current v1.0 `CellProfiler-plugins/active_plugins` location for plugins but will revert to the historical location of plugins in the `CellProfiler-plugins` root directory if the `active_plugins` folder is not present.
@@ -147,7 +159,7 @@ If you need to use deprecated plugin organization you can access previous commit
0 commit comments