docs: LLM policy by mcdurdin · Pull Request #16241 · keymanapp/keyman

mcdurdin · 2026-07-14T12:29:48Z

Create a policy on usage of LLMs, adapted from draft policy from rust-lang. Simplified and contextualized to Keyman-specific scenarios.

Test-bot: skip
Build-bot: skip

Create a policy on usage of LLMs, adapted from draft policy from rust-lang. Simplified and contextualized to Keyman-specific scenarios. Test-bot: skip Build-bot: skip

keymanapp-test-bot · 2026-07-14T12:29:56Z

User Test Results

Test specification and instructions

User tests are not required

ermshiperete

I think this is a good start. Wondering if we should distinguish between an agent (no or minimal human involved) and other LLM (that a human uses as a tool).

I'm a bit hesitant about banning all comments originally created by an LLM. I think we agree that we don't want to deal with comments from an agent and discuss code changes from/with an LLM without the human author having reviewed and understand the comments and changes.

But in our international context with a lot of non-native English speakers it could be helpful to have a LLM to draft a comment and use that as an inspiration. It should be re-stated in own words, but sometimes the wording of the LLM might be the best and I can't come up with a different wording that still communicates the point. In this case IMO it doesn't make sense to ban the comment, or even require LLM disclosure. Maybe we can somehow clarify that a bit better.

I think what we basically partially want is:

the author understands what he's writing and can explain it
the author verified the facts in the comment and changes
it's the author's reputation that is at stake, not the LLMs.

ermshiperete · 2026-07-14T13:55:15Z

+- Documentation that is originally created by an LLM.
+    - ℹ️ This includes non-trivial source comments, such as doc-comments, safety comments, or multiple paragraphs of non-doc-comments.
+    - ℹ️ This includes compiler diagnostics.
+         LLMs are conditionally allowed to assist with the *logic* surrounding a diagnostic (see "Experiment: LLM-created code changes" below),


Is "Experiment: LLM-created code changes" missing in our doc?

ermshiperete · 2026-07-14T13:56:21Z

+### Summary
+- ✅ Allowed: Private use.
+- ❌ Banned: LLM-created comments, docs, or diagnostics. Replacing human judgement with LLM judgement. Requiring people to use an LLM to contribute.
+- ⚠️ Conditionally allowed: Trivial changes, machine translation, LLM reviews and review bots, LLM-created code *under the experiment rules*.


"under the experiment rules" - can't find those

Also, to be consistent with the rest of the doc:

Suggested change

- ⚠️ Conditionally allowed: Trivial changes, machine translation, LLM reviews and review bots, LLM-created code *under the experiment rules*.

- ⚠️ Conditionally allowed: Trivial changes, machine-translation, LLM reviews and review bots, LLM-created code *under the experiment rules*.

ermshiperete · 2026-07-15T08:16:21Z

+### Legend
+- ✅ Allowed.
+- ❌ Banned.
+- ⚠️ Allowed with caveats. Must disclose that an LLM was used.
+- ℹ️ Adds additional detail to the policy. These bullets are normative.
+- 🔨 Violating this clause counts as a violation of the Code of Conduct.


Because Legend has the same heading level as the following and comes before Summary I find that a little bit hard to understand.

Wondering if something like this would be better:

Suggested change

### Legend

- ✅ Allowed.

- ❌ Banned.

- ⚠️ Allowed with caveats. Must disclose that an LLM was used.

- ℹ️ Adds additional detail to the policy. These bullets are normative.

- 🔨 Violating this clause counts as a violation of the Code of Conduct.

> [!NOTE]

> Legend:

> - ✅ Allowed.

> - ❌ Banned.

> - ⚠️ Allowed with caveats. Must disclose that an LLM was used.

> - ℹ️ Adds additional detail to the policy. These bullets are normative.

> - 🔨 Violating this clause counts as a violation of the Code of Conduct.

or

Suggested change

### Legend

- ✅ Allowed.

- ❌ Banned.

- ⚠️ Allowed with caveats. Must disclose that an LLM was used.

- ℹ️ Adds additional detail to the policy. These bullets are normative.

- 🔨 Violating this clause counts as a violation of the Code of Conduct.

Legend:

- ✅ Allowed.

- ❌ Banned.

- ⚠️ Allowed with caveats. Must disclose that an LLM was used.

- ℹ️ Adds additional detail to the policy. These bullets are normative.

- 🔨 Violating this clause counts as a violation of the Code of Conduct.

Should we cross-link to CODE_OF_CONDUCT.md somewhere?

ermshiperete · 2026-07-15T08:26:26Z

+    - ℹ️ This does not apply if the LLM content is clearly quoted and marked; you can post that.
+         However, the content of the comment must stand on its own even without the LLM content; it's not a substitute for your own words.
+    - ℹ️ See also "machine-translation" in ⚠️ below.
+    - ℹ️ See also "Scope" in the appendix below.


Suggested change

- ℹ️ See also "Scope" in the appendix below.

- ℹ️ See also ["Scope"](#scope) in the appendix below.

ermshiperete · 2026-07-15T08:28:08Z

+
+The policy's guidelines are roughly as follows:
+
+> It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to **create**.


Suggested change

> It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to **create**.

> It's fine to use LLMs to answer your own questions, analyze, distill, refine, check, suggest, review. But not to **create**.

ermshiperete · 2026-07-15T08:50:49Z

+  - Asking an LLM to summarize comments on an issue or PR.
+      - ℹ️ This does not allow reposting the summary publicly. This only includes your own personal use.
+  - Asking an LLM to privately review your code or prose.
+      - ℹ️ This does not apply to public comments by the LLM. See "review bots" under ⚠️ below.


"review bots" - we don't have a section that explicitly talks about that.

ermshiperete · 2026-07-15T08:51:01Z

+### Responsibility
+
+Your contributions are your responsibility; you cannot place any blame on an LLM.
+- ℹ️ This includes when asking people to address review comments originally created by an LLM. See "review bots" under ⚠️ above.


"review bots" - we don't have a section that explicitly talks about that.

ermshiperete · 2026-07-15T08:52:12Z

+For more background about analogous reasoning, see ["What Colour are your bits?"](https://ansuz.sooke.bc.ca/entry/23).
+
+This policy makes no distinction between LLM output that comes from a chat interface and output that comes from editor auto-completion.
+In most cases the output is "trivial" (see above under ⚠️), but regardless, it is not treated specially by this policy.


Suggested change

In most cases the output is "trivial" (see above under ⚠️), but regardless, it is not treated specially by this policy.

In most cases the output is "trivial" (see above under [⚠️](#%EF%B8%8F-allowed-with-caveats)), but regardless, it is not treated specially by this policy.

ermshiperete · 2026-07-15T08:53:07Z

+- Comments from a personal user account that are originally created by an LLM.
+    - ℹ️ This also applies to issue bodies and PR descriptions.
+    - ℹ️ This also applies to voice/video content originally created or scripted by an LLM.
+    - ℹ️ This does not apply if the LLM content is clearly quoted and marked; you can post that.
+         However, the content of the comment must stand on its own even without the LLM content; it's not a substitute for your own words.
+    - ℹ️ See also "machine-translation" in ⚠️ below.
+    - ℹ️ See also "Scope" in the appendix below.
+- Documentation that is originally created by an LLM.
+    - ℹ️ This includes non-trivial source comments, such as doc-comments, safety comments, or multiple paragraphs of non-doc-comments.
+    - ℹ️ This includes compiler diagnostics.
+         LLMs are conditionally allowed to assist with the *logic* surrounding a diagnostic (see "Experiment: LLM-created code changes" below),
+         but they must not be used to create the message itself.
+    - ℹ️ This does not include "trivial" changes (see ⚠️ below).


We list comments and documentation as banned if created by an LLM, but not code. I think we should add code changes as well.

ermshiperete · 2026-07-15T08:55:21Z

+
+### Scope
+
+This policy applies to repositories in the `keymanapp` org.


Suggested change

This policy applies to repositories in the `keymanapp` org.

This policy applies to repositories in the [`keymanapp` org](https://github.com/keymanapp).

MattGyverLee · 2026-07-17T04:46:54Z

+  - Asking an LLM to privately review your code or prose.
+      - ℹ️ This does not apply to public comments by the LLM. See "review bots" under ⚠️ below.
+  - Writing dev-tools for your own personal use using an LLM.
+  - Using an LLM to generate possible solutions to an issue, learning from them, and then writing something from scratch in your own style.


I just re-read the policy. In prose, when you want to repeat something notewrothy without plagiarism, you cite and then quote or paraphrase. But if an LLM produces the exact right contextual code fix to a problem and you can defend it, you should be able to disclose it and there's no need (and sometimes no possibility) to rewrite it in your own style. The risk isn't an AI-styled correct solution. The actual risks (as I see them) are:

With the international exposure, you end up with something clearly or subtly nefarious slipping in without review by the submitter and overworked reviewer.

The second worst risk is wasting the reviewer's time (especially for Keyman since there are too few) with cleaning something that is unreviewed by the submitter and not disclosed as unreviewed.

Code that doesn't solve the problem or makes things worse.

Code that works in isolation but doesn't account for decisions elsewhere in the codebase.

Contributor time is wasted restyling something that was already clearly good.

The crazy thing that is that those risks are exactly the same when an LLM isn't involved at all. The standard for merging AI-assisted code should be identical to human-written code: the author human must understand it, and reviewers must be satisfied with it. The rewrite process doesn't address risk one; review does. Disclosure makes that review more informed. Rewriting from scratch just adds time without adding safety.

I'd like to propose replacing this line with:

Directing an LLM to generate code, which you then review, understand, and take full responsibility for, including being able to explain any part of it in review, is allowed with disclosure.

(I deleted the duplicate comment)

There is another.
6. Code that solves the problem but makes unnecessary changes that still require the reviewer to spend precious time to verify they are not needed as it is not obvious at first.
This is not common with code not written by a LLM. That is because the LLM code seems correct at first specially with the verbose explanations around the change so the person submitting the PR believes they understand it but really they don't. You don't see this in human generated PRs(well not often).
I agree that sometimes the code is very close to as you the contributor would write it (see my comment below about creater). Therfore re-writing from scratch is over the top in some cases, but it puts the burden on the submitter to really make sure the PR is valid.

@ermshiperete I wish you hadn't deleted your comment, as you expressed some things in ways that I hadn't. I thought they were complimentary, especially since I hadn't read yours before I wrote mine.

He didn't delete his comment he deleted your verbatim duplicate comment.

yep, the exact same comment from you appeared twice, so I deleted one so that others wouldn't have to figure out that it's a duplicate... "Hide" might have been the better option than delete, sorry, didn't think of that.

rc-swag · 2026-07-21T04:20:10Z

+If you are a new contributor, you should expect to be scrutinized more heavily than existing contributors,
+since you haven't yet established trust with your reviewers.
+
+All uses under "⚠️ Allowed with caveats" **must** disclose that an LLM was used.


+1
My vote is commit message and PR description.

rc-swag · 2026-07-21T04:22:50Z

+    - ℹ️ Changes are trivial if there is no other way to write them, or the other ways to write them are nearly identical. For example, the following are all trivial:
+        - Typo fixes
+        - Markdown links
+        - Changing a word to a synonym


Human or a numerous other editing tools that are not LLM. I feel this is not really the intent of the llm policy.

rc-swag · 2026-07-21T04:43:10Z

+No amount of editing can change how it was originally created;
+the origin sets the initial style, and that style is very hard to change once it's set.
+
+For more background about analogous reasoning, see ["What Colour are your bits?"](https://ansuz.sooke.bc.ca/entry/23).


The top of this link the authour says
"As of Summer 2024, my article "What Colour are your bits?" has been online 20 years, and people are still linking to it as a benchmark. It's clear that people still care about intellectual property in general and copyright in particular, and the difference, if any, between identical copies of things is still important; but the most salient issues today are not directly related to the verbatim copying that was a big deal in 2004 and was the main topic of "What Colour." I've written other articles about today's issues, and I wish my more recent articles would get the attention that the 20-year-old one still commands; they are more relevant now.

In particular, people in 2024 care a lot about how intellectual property issues and "creator's rights" relate to material that is not actually created by humans - like the output of so-called "generative AI....."

rc-swag · 2026-07-21T05:43:44Z

+No amount of editing can change how it was originally created;
+the origin sets the initial style, and that style is very hard to change once it's set.


Ok I agree where"text that was generated..." if the text refers to documentation, code comments I agree.
For code it is not clear cut and the style is not different since the human set it.
Where was it created though?
For example the other day, I wrote a prompt that described the class I had designed with the member types and the pubic member functions. I also told it the files/classes where it would be called from. The agent basically did my biding and typed the code. I created it the LLM typed it. When I re-write it it is pretty much the same as it is my design (the code with in the functions is deterministic and usually only one way of doing it. I'm ok with the re-write. I just disagre with assumtion being made in these statements about the "original created" when it comes to code.

rc-swag · 2026-07-21T06:08:24Z

+  - Asking an LLM to privately review your code or prose.
+      - ℹ️ This does not apply to public comments by the LLM. See "review bots" under ⚠️ below.
+  - Writing dev-tools for your own personal use using an LLM.
+  - Using an LLM to generate possible solutions to an issue, learning from them, and then writing something from scratch in your own style.


There is another.
6. Code that solves the problem but makes unnecessary changes that still require the reviewer to spend precious time to verify they are not needed as it is not obvious at first.
This is not common with code not written by a LLM. That is because the LLM code seems correct at first specially with the verbose explanations around the change so the person submitting the PR believes they understand it but really they don't. You don't see this in human generated PRs(well not often).
I agree that sometimes the code is very close to as you the contributor would write it (see my comment below about creater). Therfore re-writing from scratch is over the top in some cases, but it puts the burden on the submitter to really make sure the PR is valid.

docs: LLM policy

b388866

Create a policy on usage of LLMs, adapted from draft policy from rust-lang. Simplified and contextualized to Keyman-specific scenarios. Test-bot: skip Build-bot: skip

github-project-automation Bot moved this to Todo in Keyman Jul 14, 2026

github-project-automation Bot added this to Keyman Jul 14, 2026

keymanapp-test-bot Bot added this to the A19S33 milestone Jul 14, 2026

github-actions Bot added the docs label Jul 14, 2026

ermshiperete reviewed Jul 15, 2026

View reviewed changes

MattGyverLee reviewed Jul 17, 2026

View reviewed changes

keymanapp deleted a comment from MattGyverLee Jul 17, 2026

keyman-server modified the milestones: A19S33, A19S34 Jul 20, 2026

rc-swag reviewed Jul 28, 2026

View reviewed changes

	- ⚠️ Conditionally allowed: Trivial changes, machine translation, LLM reviews and review bots, LLM-created code under the experiment rules.
	- ⚠️ Conditionally allowed: Trivial changes, machine-translation, LLM reviews and review bots, LLM-created code under the experiment rules.

-### Legend
-- ✅ Allowed.
-- ❌ Banned.
-- ⚠️ Allowed with caveats. Must disclose that an LLM was used.
-- ℹ️ Adds additional detail to the policy. These bullets are normative.
-- 🔨 Violating this clause counts as a violation of the Code of Conduct.
+> [!NOTE]
+> Legend:
+> - ✅ Allowed.
+> - ❌ Banned.
+> - ⚠️ Allowed with caveats. Must disclose that an LLM was used.
+> - ℹ️ Adds additional detail to the policy. These bullets are normative.
+> - 🔨 Violating this clause counts as a violation of the Code of Conduct.

	- ℹ️ See also "Scope" in the appendix below.
	- ℹ️ See also ["Scope"](#scope) in the appendix below.


		The policy's guidelines are roughly as follows:

		> It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to create.

	> It's fine to use LLMs to answer questions, analyze, distill, refine, check, suggest, review. But not to create.
	> It's fine to use LLMs to answer your own questions, analyze, distill, refine, check, suggest, review. But not to create.

	In most cases the output is "trivial" (see above under ⚠️), but regardless, it is not treated specially by this policy.
	In most cases the output is "trivial" (see above under [⚠️](#%EF%B8%8F-allowed-with-caveats)), but regardless, it is not treated specially by this policy.


		### Scope

		This policy applies to repositories in the `keymanapp` org.

		No amount of editing can change how it was originally created;
		the origin sets the initial style, and that style is very hard to change once it's set.

Uh oh!

Uh oh!

Conversation

mcdurdin commented Jul 14, 2026

Uh oh!

keymanapp-test-bot Bot commented Jul 14, 2026

User Test Results

Uh oh!

ermshiperete left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ermshiperete Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ermshiperete Jul 29, 2026 •

edited

Loading