Add framesets (.dtas)

[rpguiteras]

Add support for Stata framesets (.dtas).

The main substantive changes are in pypkg/src/pystatacons/stata_utils.py and src/complete_datasignature.ado.

Checksum for .dtas is concatenated checksums of individual frames, in alphabetical order, for example

output/data/project-area.dtas: LGD_district=23:4(30770):1387127574:1572535773:1213271928|LGD_subdistrict=97:9(94209):517290162:1301972081:2412684702

Ignores the order in which frames are saved within frameset, ignores frame's date characteristic.

statacons.ado and stataconsign.ado are not substantively altered (just bump version number):

Also added examples in frames/examples and tests in frames/tests.

Still works on Stata 16 and 17 even though these cannot use framesets (.dtas) -- uses default scons checksum for .dtas. (Works in the sense that statacons will parse things properly, of course code using .dtas with Stata < 18 will fail -- gives warnings.)

[bquistorff]

A few comments:

1. We should avoid committing data files when possible, so I'm concerned about the files in frames/datasets. Can we generate what we need for testing/docs on the fly using sysuse datasets?
2. For complete_datasignature.ado in interactive mode, will the frame in use before the call to complete_datasignature be the same after the round-trip to the frames file?
3. There can be many frames in dtas file and so the signature can grow quite long. Are there any signature length limits in cons that we should be aware of? If so, we could do a checksum of the long signature like I did in the dtas branch.

Other than that, I can test the frames code, so I'm assuming there's sufficient tests and that they pass on your machine.

[rpguiteras]

Ideas not implemented:

Allow SConscripts to refer to specific frames. Currently if one frame changes, the signature for the .dtas as a whole changes, so everything downstream is rebuilt. However, some tasks might use only a subset of frames and not need rebuilding if a different frame in the frameset changes. Since the signature for the .dtas concatenates signatures of individual frames, in principle this should be possible.

Stata 16 and 17 could unzip the .dtas file and calculate complete datasignatures for each component .dta file that does not contain any alias variables. 16/17 could not open a .dta file with alias variables, so we would have to fall back to a default MD5.

[rpguiteras]

A few comments:

1. We should avoid committing data files when possible, so I'm concerned about the files in frames/datasets. Can we generate what we need for testing/docs on the fly using sysuse datasets?

Yes, these are all sysuse or webuse. the do-file _refresh_datasets.do grabs all the webuse ones.

I wasn't too worried about keeping them on the repo since they are <0.5 MB total, but we could .gitignore them.

2. For complete_datasignature.ado in interactive mode, will the frame in use before the call to complete_datasignature be the same after the round-trip to the frames file?

Claude says no, and suggests a fix. I will give it a try and run the tests.

 - No --- the current frame is not restored. Here is why:

  The round-trip (lines 29 and 51) saves with frames save, frames(_all) and restores with frames use, clear. Neither command records
  nor replays which frame was active at save time. The .dtas format's .frameinfo manifest stores frame names and DTA filenames, but not
   the "current frame" pointer. After frames use the active frame is default (Stata's reset behavior on load), regardless of what the
  user was in before.

  The fix is to capture c(frame) before the save and frame change back after the restore:

  if !`is_batch' {
      loc saved_frame "`c(frame)'"          // <-- add
      qui frames save "`tempdtas'", frames(_all) replace emptyok
      loc need_restore = 1
  }

  if `need_restore' {
      qui frames use "`tempdtas'", clear
      cap erase "`tempdtas'"
      qui frame change `saved_frame'         // <-- add
  }

  The frame change should be guarded (e.g., cap) in case the frame name that was active before happens not to exist in the restored
  frameset, though for a well-formed user session that situation shouldn't arise.

Copilot (GPT 5.4) also says no:

Not guaranteed by complete_datasignature.ado itself.

  In the interactive frameset_file() path, the ado saves all frames to a temp .dtas, loads the target frameset, computes signatures, then restores with:

   - frames save "tempdtas'", frames(_all) ...`
   - frames use "tempdtas'", clear`

  That restores the set of frames and their contents, but the code never records the pre-call active frame name and never does a final frame change <oldframe> afterward. So the frame you end up in after the restore depends entirely on how frames use chooses the active frame when loading 
  the temp frameset, not on an explicit restoration step in the ado.

  So the safe answer is: the user state is restored, but the previously active frame is not explicitly preserved.

3. There can be many frames in dtas file and so the signature can grow quite long. Are there any signature length limits in cons that we should be aware of? If so, we could do a checksum of the long signature like I did in the dtas branch.

Gemini says no

Custom Signatures and Database Storage

If you implement a custom Decider function in your SConstruct file, you can technically return a custom string or object to serve as the "signature" for a file.

Storage via Pickle: SCons saves its tracking data inside a database file (usually .sconsign.dblite). It handles this data using Python’s standard pickle serialization module.

No Arbitrary Cap: Because pickle stores arbitrary Python objects and strings natively, SCons will not reject a custom signature string just because it is exceptionally long. Your only real boundary is your system's available RAM and Python's theoretical string limit ($2^{31}-1$ bytes on 32-bit systems or $2^{63}-1$ bytes on 64-bit systems).

Other than that, I can test the frames code, so I'm assuming there's sufficient tests and that they pass on your machine.

Yes, these passed on Stata17 and 19 on my PC and Stata 19 on our cluster (only tested batch on the cluster I think).

It also worked well in an "out of sample" test - a project where I had written up an SConscript with framesets (.dtas), but was not consulted in designing the new code

[rpguiteras]

I implemented a fix for your comment 2 above and ran the interactive and batch mode tests again successfully.

This is in a new commit 683c8f5, with tag 3.1.0-alpha3.

I'm not sure what the etiquette is, do I close this PR and open a new one from that commit? Actually it looks like those commits show up in here so I guess no action needed?

[rpguiteras]

@bquistorff should I have created a fork instead of a branch? So we can merge in milestone versions before publishing 3.1.0? If you think this is close to publication-ready then we can just work here, I guess, and I'll do a branch for future work (tutorial, frames within framesets).