fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! fixup! [feature] building up the schema again from scratch acording the section 11,4,1 in the bitag report

Author: marmoure

src/main/resources/schemas/sample.csv

@@ -1,6 +1,6 @@
-date_published,regulator_name,regulator_version_number,connection_type,fcc_registration_number,unique_plan_identifier,network_technology_type,provider_name,service_plan_name,bandwidth_download_units,bandwidth_download_marketed_low,bandwidth_download_marketed_high,bandwidth_download_typical_low,bandwidth_download_typical_high,bandwidth_upload_units,bandwidth_upload_marketed_low,bandwidth_upload_marketed_high,bandwidth_upload_typical_low,bandwidth_upload_typical_high,latency_idle_low,latency_idle_high,currency,price_type,price_details,price_recurring,fee_introductory,fee_introductory_description,contract_required,contract_details,contract_terms_uri,fee_one_time_amount,fee_one_time_details,fee_recurring,fee_recurring_description,fee_pass_through_recurring,fee_pass_through_recurring_description,fee_early_termination,fee_early_termination_description,government_taxes,policy_data_usage,fee_data_usage,fee_data_usage_description,uri_data_usage,restrictions_apply,restrictions_description,uri_policy_network_management,uri_policy_privacy,uri_customer_support,customer_support_phone,country_code
-2025-01-15,FCC,1,fixed,0005937974,F0005937974TIER1,50,Xfinity Internet,Extreme 500,Mbps,2,2,2,2,Mbps,2,2,2,2,2,2,USD,Pre-Paid,Requires activation of one new device.,100,10.32,only for new customers,No,Contract is 1 year.,https://example.com/contract-terms/internet/,29,Professional installation fee of $100 required.,," Monthly device rental."," $12.99"," Emergency services cost recovery fee",0," $100 charged if service is terminated before the end of one year.",Included,Yes,,,https://example.com/data-usage/internet/,Yes,Site survey required,https://example.com/network-management/internet/,https://example.com/privacy/internet/,https://example.com/support.html,18005551212,USA
-2025-11-02,,1,mobile,0003768165,F0005937974ABCD12341234,500,T-Mobile,Turbo 5G Plus,Mbps,75,75,75,75,Mbps,75,75,75,75,75,75,USD,Post-Paid,Price covers first 1 GB of data transferred. Data transfer billed at $1 per GB thereafter.,100,10.32,only for new customers,No,Valid for first 3 months.,https://example.com/internet-contract/,29.99,Shipping fee of $20 for sending equipment.,9,Cloud storage fee.,$10,State regulatory assessment fee,100,$25,Varies,Yes,1,$1 per GB after the first 1 TB.,https://example.com/datausagepolicies,Yes,Qualified for a low-income offering,https://example.com/networkpolicies,https://example.com/privacy-policies/,https://example.com/support/internet/,18885551212,CAN
+date_published,regulator_name,regulator_version_number,connection_type,fcc_registration_number,unique_plan_identifier,network_technology_type,provider_name,service_plan_name,bandwidth_download_units,bandwidth_download_marketed_low,bandwidth_download_marketed_high,bandwidth_download_typical_low,bandwidth_download_typical_high,bandwidth_upload_units,bandwidth_upload_marketed_low,bandwidth_upload_marketed_high,bandwidth_upload_typical_low,bandwidth_upload_typical_high,latency_idle_low,latency_idle_high,currency,price_type,price_details,price_recurring,fee_introductory,fee_introductory_description,contract_required,contract_details,contract_terms_uri,fee_one_time_amount,fee_one_time_details,fee_recurring,fee_recurring_description,fee_pass_through_recurring,fee_pass_through_recurring_description,fee_early_termination,fee_early_termination_description,government_taxes,policy_data_usage,fee_data_usage,fee_data_usage_description,uri_data_usage,restrictions_apply,restrictions_description,uri_policy_network_management,uri_policy_privacy,uri_customer_support,customer_support_phone,country_code,asn,uri_policy_additional_terms,language,digital_signature
+2025-01-15,FCC,1,fixed,0005937974,F0005937974TIER1,50,Xfinity Internet,Extreme 500,Mbps,2,2,2,2,Mbps,2,2,2,2,2,2,USD,Pre-Paid,Requires activation of one new device.,100,10.32,only for new customers,No,Contract is 1 year.,https://example.com/contract-terms/internet/,29,Professional installation fee of $100 required.,," Monthly device rental."," $12.99"," Emergency services cost recovery fee",0," $100 charged if service is terminated before the end of one year.",Included,Yes,,,https://example.com/data-usage/internet/,Yes,Site survey required,https://example.com/network-management/internet/,https://example.com/privacy/internet/,https://example.com/support.html,18005551212,USA,,,,
+2025-11-02,,1,mobile,0003768165,F0005937974ABCD12341234,500,T-Mobile,Turbo 5G Plus,Mbps,75,75,75,75,Mbps,75,75,75,75,75,75,USD,Post-Paid,Price covers first 1 GB of data transferred. Data transfer billed at $1 per GB thereafter.,100,10.32,only for new customers,No,Valid for first 3 months.,https://example.com/internet-contract/,29.99,Shipping fee of $20 for sending equipment.,9,Cloud storage fee.,$10,State regulatory assessment fee,100,$25,Varies,Yes,1,$1 per GB after the first 1 TB.,https://example.com/datausagepolicies,Yes,Qualified for a low-income offering,https://example.com/networkpolicies,https://example.com/privacy-policies/,https://example.com/support/internet/,18885551212,CAN,,,,
 2025-06-06,FCC,99,satellite,0005937974,F0005937974ABCDEFGHIJ12345,500,Google Fiber,Fiber 2G,Gbps,500,500,500,500,Gbps,500,500,500,500,500,500,USD,Volume-Based,For new customers only.,100,10.32,only for new customers,Yes,"Contract must be digitally signed within 7 days of
 installation.",https://example.com/legal/terms/contracts.html,399.00,"$10
 device provisioning fee.
@@ -9,5 +9,5 @@ Street USA;Local franchise fee for Gotham City
 ",9.99,"$10 for every month
 remaining in contract term.",Not Applicable,No,,"No fee but bitrates are reduced to 100 Mbps after 10 GB of
 usage.",https://example.com/data-usage.html,No,"Mobile service only available to existing
-customers of wireline service",https://example.com/network-management.html,https://example.com/privacypolicy.html,https://example.com/support,12675551212,JPN
-2025-06-06,FCC,,something,,,0,randomeValue,,kbps,1,1,1,1,kbps,1,1,1,1,1,1,USD,Other,Requires activation of one new device.,100,10.32,only for new customers,No,,,,,12.99,,,,0,,Included,No,143.32,$2 per GB during peak times and $1 per GB during off-peak times.,,Yes,,,,,,
+customers of wireline service",https://example.com/network-management.html,https://example.com/privacypolicy.html,https://example.com/support,12675551212,JPN,,,,
+2025-06-06,FCC,,something,,,0,randomeValue,,kbps,1,1,1,1,kbps,1,1,1,1,1,1,USD,Other,Requires activation of one new device.,100,10.32,only for new customers,No,,,,,12.99,,,,0,,Included,No,143.32,$2 per GB during peak times and $1 per GB during off-peak times.,,Yes,,,,,,,,,,

src/main/resources/schemas/schema.csvs

@@ -1,5 +1,5 @@
 version 1.1
-@totalColumns 50
+@totalColumns 54
 date_published: xDate
 regulator_name: empty or upperCase
 regulator_version_number: empty or range(1, 99)
@@ -50,3 +50,7 @@ uri_policy_privacy: empty or uri
 uri_customer_support: empty or uri
 customer_support_phone: empty or regex("^[0-9]{7,15}$") // TODO: min length for phone number is 7
 country_code: empty or regex("^[A-Z]{3}$") @optional
+asn: empty or length(0,4294967295) @optional
+uri_policy_additional_terms: empty or uri @optional
+language: empty or length(3) @optional
+digital_signature: empty or length(0, 500) @optional

src/main/resources/schemas/schema_specs

@@ -555,4 +555,50 @@ than the two-letter code. For example, the United States of America is USA and C
 This field MUST be in the file and MAY be populated.
 Name: country_code
 Format/Type: string, length(3)
-Examples: USA, CAN, JPN
+Examples: USA, CAN, JPN
+
+51. Autonomous System Number (ASN).
+It may be useful to include a provider’s ASN in the file. One example of this may be that a researcher is
+collecting network measurements using source IP address, which can be used to identify the ASN, or ASN is
+part of the collected data (such as related to the Border Gateway Protocol, BGP, for routing security to see
+if a Route Object Authorization, ROA, was issued), making it easy for a researcher to find labels based on
+the ASN or to combine label files with their data.
+A complicating factor can be that some ISPs have several ASNs, though often these aggregate up into one
+primary ASN. In other cases, small ISPs may not have their own ASN and thus may be using the ASN of
+their upstream provider. For these reasons, this field is considered optional. Thus, if the ISP does not have
+their own ASN, it SHOULD be left empty. If an ISP has multiple ASNs, then the primary or aggregate ASN
+SHOULD be used, or this field MAY be left empty.
+ASN as maintained by the Internet Assigned Numbers Authority (IANA). [46] In North America, these
+are issued by the American Registry for Internet Numbers (ARIN). [47] For US ISPs, the ARIN ASN list
+shall be used. [48] Until 2007, ASN were defined as 16-bit integers, which allowed for a maximum of 65,536
+assignments. But the IANA is now assigning 32-bit ASNs up to 4,294,967,295. As a result this field will be
+numeric and limited to 10 digits in length.
+This field MUST be in the file and MAY be populated.
+Name: asn
+Format/Type: string, length(0,4294967295)
+Examples: 7922, 14593, 701
+
+
+52. Additional Terms Uniform Resource Indicator (URI).
+This is the URI for any additional terms and/or policies that may apply to a specific BIAS offering. If
+nothing applies, this field should be left empty. If multiple policies apply, then URIs MUST be separated by
+a semi-colon.
+This field MUST be in the file and MAY be populated.
+Name: uri_policy_additional_terms
+Type/Format: identifier, uri
+Examples: https://www.example.com/terms-of-service/, https://www.example.com/terms/;https://www.example.com/abuse/https://www.example.com/anti-abuse/
+
+53. Language.
+This field can optionally specify the language used in the machine-readable file. The default language shall
+be English. This field will use the ISO 639-3 format [49], which is a three letter format
+This field MUST be in the file and MAY be populated.
+Name: language
+Type/Format: string, length(3)
+Examples: JPN, DEU, ITA
+
+54. Digital Signature.
+If, in the future, labels are individually cryptographically signed by providers, the signature string can be
+inserted into this field.
+Name: digital_signature
+Type/Format: string
+Examples: None at this time