[Yaml-core] Pygments YAML Lexer (from Spec Update)

[Jeff Kowalczyk]

(imminent) updated 1.1 YAML spec containing the updated productions.

As the executable spec update gets finalized, would it be possible to emit
productions in a form that would make a YAML lexer for Pygments easy,
perhaps trivial to implement?

http://pygments.org/docs/lexerdevelopment/

YAML support in Pygments would be of relatively high value to YAML, since
it is the default syntax highlighter in the next version of the popular
Project/SCM package Trac.

Sidebar: would Pygments RegexLexer be sufficient to lex YAML syntax?

Without much parser knowledge, from this list's discussions I've come to
expect that a coloring YAML lexer might not need the indentation
state/context as would be required by YAML parser.

Thanks.

[Oren Ben-Kiki]

This looks very interesting, and yes potentially it would be possible to
plug the productions into it. The main lack I see in the framework there is
that it doesn't support passing parameters to a 'state' or 'production', at
least not "out of the box". However there seems to be multiple ways this
could be added into the system, ranging from a global stack to using
closures. I haven't really coded in Python so I have no idea what would be
the best approach. It certainly should be doable...

I have been caught up in a work project again, but hopefully next week I'll
be able to upload the existing (Haskell) code which you could use as an
inspiration.

Have fun,

Oren Ben-Kiki

Post by Jeff Kowalczyk

(imminent) updated 1.1 YAML spec containing the updated productions.

As the executable spec update gets finalized, would it be possible to emit
productions in a form that would make a YAML lexer for Pygments easy,
perhaps trivial to implement?
http://pygments.org/docs/lexerdevelopment/

[Jeff Kowalczyk]

Resurrecting an old thread. Can we get a URL-reachable Reference.bnf or
another plaintext representation of the spec productions put up and kept
updated on yaml.org?

There are enough developers adopting sphinx+pygments that someone is
likely to find the YAML BNF useful enough to consume for a colorizing
lexer.

Cool move writing the yaml.org homepage in YAML. With a pygments YAML
colorizer, it might look a bit prettier, too ;)

Thanks,
Jeff

Post by Oren Ben-Kiki
This looks very interesting, and yes potentially it would be possible to
plug the productions into it. The main lack I see in the framework there is
that it doesn't support passing parameters to a 'state' or 'production', at
least not "out of the box". However there seems to be multiple ways this
could be added into the system, ranging from a global stack to using
closures. I haven't really coded in Python so I have no idea what would be
the best approach. It certainly should be doable...
I have been caught up in a work project again, but hopefully next week I'll
be able to upload the existing (Haskell) code which you could use as an
inspiration.
Have fun,
Oren Ben-Kiki

Post by Jeff Kowalczyk

(imminent) updated 1.1 YAML spec containing the updated productions.

As the executable spec update gets finalized, would it be possible to emit
productions in a form that would make a YAML lexer for Pygments easy,
perhaps trivial to implement?
http://pygments.org/docs/lexerdevelopment/

[Kirill Simonov]

Hi Jeff,

Here is my attempt to write a YAML lexer for Pygments. It's hardly
tested, so expect bugs. I was also unable to choose suitable token
types from the existing tokens provided by the package, so I had to
create YAML-specific tokens.

The lexer file as well as a yaml file I've used for tests are here:
http://svn.pyyaml.org/pyyaml/trunk/examples/pygments-lexer/

Thanks,
Kirill

[Jeff Kowalczyk]

Post by Kirill Simonov
Here is my attempt to write a YAML lexer for Pygments. It's hardly
tested, so expect bugs. I was also unable to choose suitable token
types from the existing tokens provided by the package, so I had to
create YAML-specific tokens.
http://svn.pyyaml.org/pyyaml/trunk/examples/pygments-lexer/
Thanks,
Kirill

Thank you very much for the implementation, Kirill. I've integrated the
code as a patch to pygments/lexers/text.py, and bundled it for upstream to
include in pygments:

http://dev.pocoo.org/projects/pygments/ticket/328

Thanks,
Jeff

[Jeff Kowalczyk]

Post by Jeff Kowalczyk

Post by Kirill Simonov
Here is my attempt to write a YAML lexer for Pygments. It's hardly
tested, so expect bugs. I was also unable to choose suitable token
types from the existing tokens provided by the package, so I had to
create YAML-specific tokens.
http://svn.pyyaml.org/pyyaml/trunk/examples/pygments-lexer/
Thanks,
Kirill

Thank you very much for the implementation, Kirill. I've integrated the
code as a patch to pygments/lexers/text.py, and bundled it for upstream to
http://dev.pocoo.org/projects/pygments/ticket/328
Thanks,
Jeff

I'm happy to report that Kirill's YamlLexer implementation was accepted
right away into pygments main mercurial repository.

Changeset [564:3c18200871c5] by gbrandl Rename YAMLLexer to YamlLexer? to
be consistent with other names.
http://dev.pocoo.org/projects/pygments/changeset/564%3A3c18200871c5

Changeset [563:cfe7884f4d73] by gbrandl Merge after yaml lexer addition.
http://dev.pocoo.org/projects/pygments/changeset/563%3Acfe7884f4d73

YamlLexer should be present in the next release of pygments, which means
tools like Trac and Sphinx will soon colorize YAML.

gbrandl made some token=style mapping substitutions for us. Some YAML
tokens are still custom style names, and as such are not visibly colorized
out of the box (the user would need to supply a CSS rule).

I'll gather some information on all pygments built-in styles and start a
new thread here to get some input on additional syntax tokens which could
reasonably map to built-in styles.

Thanks again Kirill,
Jeff

[Jeff Kowalczyk]

Post by Jeff Kowalczyk
I'm happy to report that Kirill's YamlLexer implementation was accepted
right away into pygments main mercurial repository.

See the current output at:

http://pygments.org/demo/?lang=yaml&sort=

YamlLexer example.yaml http://pygments.org/demo/810

YAML.org homepage YAML http://pygments.org/demo/809

View HTML source to see the currently-unstyled CSS class id's, e.g:

<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">

[Kirill Simonov]

Hi Jeff,

Post by Jeff Kowalczyk

Post by Jeff Kowalczyk
I'm happy to report that Kirill's YamlLexer implementation was accepted
right away into pygments main mercurial repository.

http://pygments.org/demo/?lang=yaml&sort=
YamlLexer example.yaml http://pygments.org/demo/810
YAML.org homepage YAML http://pygments.org/demo/809
<span class="p-Indicator">-</span> <span class="l-Scalar-Plain">

Thank you for your efforts, Jeff!

I've noticed that plain scalars within flow collections are highlighted,
but those outside [] or {} are not. This is because for
'plain-scalar-in-flow-context', the class Name.Variable is used while
'plain-scalar-in-block-context' still uses Literal.Scalar.Plain. Also
escape codes got highlighted with String.Escape, but escape characters
didn't.

Personally, I'll prefer to have all scalar text not to be highlighted at
all. Instead it would be most useful if the punctuation marks
(indicators) are highlighted so that the document structure is emphasized.


Thanks,
Kirill