Adding a Scintilla based control to Xojo

Christian_Schmitz · December 7, 2021, 1:38pm

In the last years we saw a lot of need for various edit controls in Xojo. The built-in TextArea is fine for a lot of people, but some need more. A couple of people extended the built-in control with various extensions, but those are always limited by what the platform can do. MacOS is quite good with NSTextView used as implementation and you see a lot of stuff via our NSTextViewMBS class to e.g. insert a picture. On Windows the RichEdit control is a bit limited, but we still got a few TextArea extensions in our plugins:

Next a few people started to write Canvas based control on their own. Like the Formatted Text Control for Xojo, which is now open source and can be used as an editor for styled text. Then recently we got notice of the Better Code Editor (BCE) Control from Garry Pettet. This may also help a few people and have it as source code in their projects and modify it.

But I looked for more and thanks to a little tip from Norman on my visit in September, I got pointed to the Scintilla project. That is am open source C++ library defining an editing component, which we can host in a plugin based Xojo control: ScintillaControlMBS
The description from their website shows what we may get in Xojo:

As well as features found in standard text editing components, Scintilla includes features especially useful when editing and debugging source code. These include support for syntax styling, error indicators, code completion and call tips. The selection margin can contain markers like those used in debuggers to indicate breakpoints and the current line. Styling choices are more open than with many editors, allowing the use of proportional fonts, bold and italics, multiple foreground and background colours and multiple fonts.

We may not get all things right into the control for the first release, but since the control seems to work fine for macOS and Windows already, I am confident we can ship this as a 1.0 with our next plugin release in January. There is still a long todo list, but we like to give interested developers a preview. Please try it in next pre-release.

Things we may do later, if possible include support for Linux. Maybe we can provide the library within the plugin instead of an external library. And we may improve the API to be more Xojo like, add printing functions, marker pictures or async loading.

This control should provide a great editor for XojoScript code in various projects. Also editing other things like SQL statements, JavaScript, JSON or HTML in your project. It has a great performance and should handle huge texts better than all the other choices.

Please try it in the next days and let us know what you think!

Christian_Schmitz · December 7, 2021, 1:39pm

you can get pr2 here: MBS Xojo Plugins, version 21.6pr2

Jose_Cuevas · December 28, 2021, 6:49am

@Christian_Schmitz , I started playing with scintilla, Do you expose any methods to send basic messages directly to scintilla, mainly:

sptr_t ScintillaCocoa::WndProc(Message iMessage, uptr_t wParam, sptr_t lParam)
Editor::StyleSetMessage(Message iMessage, uptr_t wParam, sptr_t lParam)

Also to allow multiple buffers/documents (tabs) we would need access to a whole lot more stuff and primitives.

(PS: back in the days somebody had ported Scintilla to RealBasic, I’m going to dig in my old hard drives, besides this one for windows there was one for mac.)

Christian_Schmitz · December 28, 2021, 9:32am

While I could expose the direct function, it’s much too easy to cause crashes with passing invalid parameters.

So please let us know what functionality you need and we may add function for that.

James_Pitchford · February 1, 2022, 3:29pm

Scintilla looks v. interesting. I’ve downloaded latest example on MBS. Questions:

• how to initialise other libraries? Is it necessary to add all the keywords, styles, markers etc per your other examples or are these already included in the libraries?
• what other libraries are available?

Christian_Schmitz · February 1, 2022, 6:23pm

You talk about lexers, I presume.

here is the list:

a68k, abaqus, ada, apdl, as, asm, asn1, asy, au3, ave, avs, baan, bash, batch, bib, blitzbasic, bullant, caml, cil, clarion, clarionnocase, cmake, COBOL, coffeescript, conf, cpp, cppnocase, csound, css, d, dataflex, diff, DMAP, DMIS, ecl, edifact, eiffel, eiffelkw, erlang, errorlist, escript, f77, flagship, forth, fortran, freebasic, fsharp, gap, gui4cli, haskell, hollywood, hypertext, ihex, indent, inno, json, julia, kix, kvirc, latex, lisp, literatehaskell, lot, lout, lua, magiksf, makefile, markdown, matlab, maxima, metapost, mmixal, modula, mssql, mysql, nim, nimrod, nncrontab, nsis, null, octave, opal, oscript, pascal, powerbasic, perl, phpscript, PL/M, po, pov, powerpro, powershell, abl, props, ps, purebasic, python, r, raku, rebol, registry, ruby, rust, sas, scriptol, smalltalk, SML, sorcins, specman, spice, sql, srec, stata, fcST, TACL, tads3, TAL, tcl, tcmd, tehex, tex, txt2tags, vb, vbscript, verilog, vhdl, visualprolog, x12, xml, yaml, xojo

Next to the example project is a constants file with all the constants for the various style classes used by those lexers.

James_Pitchford · February 1, 2022, 8:13pm

Christian, thanks, looks very extensive, but sorry if I’m being a bit slow here.

So for example for HTML - what are the steps to configure the lexer?

In your example I can see:

c.InitializeLexer “hypertext” //assume hypertext from the list.

But then I don’t see constants associated with HTML or Hypertext. And further, do I need to then define the colors, keywords, styles and markers - or are they already defined within the lexer?

Christian_Schmitz · February 1, 2022, 8:46pm

First please use the version from 22.1pr1 with a few bug fixes.

Then check the “LexerConstants.txt” file we include.
It contains the nearly 2000 constants used by the lexers.

The lexer for html is named “hypertext”, but can also be used as “xml” for XML.

James_Pitchford · February 2, 2022, 1:57pm

OK, so I’m still a bit slow.

Having initialised the lexer, I still need to use the constants (or integers) to define the colours for each tag type, plus I need to define the keywords to highlight for each language I use, is that correct?

Effectively I have to reproduce everything you have in your MySQL and XojoExample methods for each language?

Christian_Schmitz · February 2, 2022, 4:24pm

Okay, I created you this list from C for you:

const SCE_H_DEFAULT = 0	// default	Text
const SCE_H_TAG = 1	// tag	Tags
const SCE_H_ERRORTAGUNKNOWN = 2	// error tag	Unknown Tags
const SCE_H_ATTRIBUTE = 3	// attribute	Attributes
const SCE_H_ATTRIBUTEUNKNOWN = 4	// error attribute	Unknown Attributes
const SCE_H_NUMBER = 5	// literal numeric	Numbers
const SCE_H_DOUBLESTRING = 6	// literal string	Double quoted strings
const SCE_H_SINGLESTRING = 7	// literal string	Single quoted strings
const SCE_H_OTHER = 8	// tag operator	Other inside tag, including space and '='
const SCE_H_COMMENT = 9	// comment	Comment
const SCE_H_ENTITY = 10	// literal	Entities
const SCE_H_TAGEND = 11	// tag	XML style tag ends '/>'
const SCE_H_XMLSTART = 12	// identifier	XML identifier start '<?'
const SCE_H_XMLEND = 13	// identifier	XML identifier end '?>'
const SCE_H_SCRIPT = 14	// error	Internal state which should never be visible
const SCE_H_ASP = 15	// preprocessor	ASP <% ... %>
const SCE_H_ASPAT = 16	// preprocessor	ASP <% ... %>
const SCE_H_CDATA = 17	// literal	CDATA
const SCE_H_QUESTION = 18	// preprocessor	PHP
const SCE_H_VALUE = 19	// literal string	Unquoted values
const SCE_H_XCCOMMENT = 20	// comment	JSP Comment <%-- ... --%>
const SCE_H_SGML_DEFAULT = 21	// default	SGML tags <! ... >
const SCE_H_SGML_COMMAND = 22	// preprocessor	SGML command
const SCE_H_SGML_1ST_PARAM = 23	// preprocessor	SGML 1st param
const SCE_H_SGML_DOUBLESTRING = 24	// literal string	SGML double string
const SCE_H_SGML_SIMPLESTRING = 25	// literal string	SGML single string
const SCE_H_SGML_ERROR = 26	// error	SGML error
const SCE_H_SGML_SPECIAL = 27	// literal	SGML special (#XXXX type)
const SCE_H_SGML_ENTITY = 28	// literal	SGML entity
const SCE_H_SGML_COMMENT = 29	// comment	SGML comment
const SCE_H_SGML_1ST_PARAM_COMMENT = 30	// error comment	SGML first parameter - lexer internal. It is an error if any text is in this style.
const SCE_H_SGML_BLOCK_DEFAULT = 31	// default	SGML block
const SCE_HJ_START = 40	// client javascript default	JS Start - allows eol filled background to not start on same line as SCRIPT tag
const SCE_HJ_DEFAULT = 41	// client javascript default	JS Default
const SCE_HJ_COMMENT = 42	// client javascript comment	JS Comment
const SCE_HJ_COMMENTLINE = 43	// client javascript comment line	JS Line Comment
const SCE_HJ_COMMENTDOC = 44	// client javascript comment documentation	JS Doc comment
const SCE_HJ_NUMBER = 45	// client javascript literal numeric	JS Number
const SCE_HJ_WORD = 46	// client javascript identifier	JS Word
const SCE_HJ_KEYWORD = 47	// client javascript keyword	JS Keyword
const SCE_HJ_DOUBLESTRING = 48	// client javascript literal string	JS Double quoted string
const SCE_HJ_SINGLESTRING = 49	// client javascript literal string	JS Single quoted string
const SCE_HJ_SYMBOLS = 50	// client javascript operator	JS Symbols
const SCE_HJ_STRINGEOL = 51	// client javascript error literal string	JavaScript EOL
const SCE_HJ_REGEX	client javascript literal regex = 52	// JavaScript RegEx	
const SCE_HJA_START = 55	// server javascript default	JS Start - allows eol filled background to not start on same line as SCRIPT tag
const SCE_HJA_DEFAULT = 56	// server javascript default	JS Default
const SCE_HJA_COMMENT = 57	// server javascript comment	JS Comment
const SCE_HJA_COMMENTLINE = 58	// server javascript comment line	JS Line Comment
const SCE_HJA_COMMENTDOC = 59	// server javascript comment documentation	JS Doc comment
const SCE_HJA_NUMBER = 60	// server javascript literal numeric	JS Number
const SCE_HJA_WORD = 61	// server javascript identifier	JS Word
const SCE_HJA_KEYWORD = 62	// server javascript keyword	JS Keyword
const SCE_HJA_DOUBLESTRING = 63	// server javascript literal string	JS Double quoted string
const SCE_HJA_SINGLESTRING = 64	// server javascript literal string	JS Single quoted string
const SCE_HJA_SYMBOLS = 65	// server javascript operator	JS Symbols
const SCE_HJA_STRINGEOL = 66	// server javascript error literal string	JavaScript EOL
const SCE_HJA_REGEX = 67	// server javascript literal regex	JavaScript RegEx
const SCE_HB_START = 70	// client basic default	Start
const SCE_HB_DEFAULT = 71	// client basic default	Default
const SCE_HB_COMMENTLINE = 72	// client basic comment line	Comment
const SCE_HB_NUMBER = 73	// client basic literal numeric	Number
const SCE_HB_WORD = 74	// client basic keyword	KeyWord
const SCE_HB_STRING = 75	// client basic literal string	String
const SCE_HB_IDENTIFIER = 76	// client basic identifier	Identifier
const SCE_HB_STRINGEOL = 77	// client basic literal string	Unterminated string
const SCE_HBA_START = 80	// server basic default	Start
const SCE_HBA_DEFAULT = 81	// server basic default	Default
const SCE_HBA_COMMENTLINE = 82	// server basic comment line	Comment
const SCE_HBA_NUMBER = 83	// server basic literal numeric	Number
const SCE_HBA_WORD = 84	// server basic keyword	KeyWord
const SCE_HBA_STRING = 85	// server basic literal string	String
const SCE_HBA_IDENTIFIER = 86	// server basic identifier	Identifier
const SCE_HBA_STRINGEOL = 87	// server basic literal string	Unterminated string
const SCE_HP_START = 90	// client python default	Embedded Python
const SCE_HP_DEFAULT = 91	// client python default	Embedded Python
const SCE_HP_COMMENTLINE = 92	// client python comment line	Comment
const SCE_HP_NUMBER = 93	// client python literal numeric	Number
const SCE_HP_STRING = 94	// client python literal string	String
const SCE_HP_CHARACTER = 95	// client python literal string character	Single quoted string
const SCE_HP_WORD = 96	// client python keyword	Keyword
const SCE_HP_TRIPLE = 97	// client python literal string	Triple quotes
const SCE_HP_TRIPLEDOUBLE = 98	// client python literal string	Triple double quotes
const SCE_HP_CLASSNAME = 99	// client python identifier	Class name definition
const SCE_HP_DEFNAME = 100	// client python identifier	Function or method name definition
const SCE_HP_OPERATOR = 101	// client python operator	Operators
const SCE_HP_IDENTIFIER = 102	// client python identifier	Identifiers
const SCE_HPHP_COMPLEX_VARIABLE = 104	// server php identifier	PHP complex variable
const SCE_HPA_START = 105	// server python default	ASP Python
const SCE_HPA_DEFAULT = 106	// server python default	ASP Python
const SCE_HPA_COMMENTLINE = 107	// server python comment line	Comment
const SCE_HPA_NUMBER = 108	// server python literal numeric	Number
const SCE_HPA_STRING = 109	// server python literal string	String
const SCE_HPA_CHARACTER = 110	// server python literal string character	Single quoted string
const SCE_HPA_WORD = 111	// server python keyword	Keyword
const SCE_HPA_TRIPLE = 112	// server python literal string	Triple quotes
const SCE_HPA_TRIPLEDOUBLE = 113	// server python literal string	Triple double quotes
const SCE_HPA_CLASSNAME = 114	// server python identifier	Class name definition
const SCE_HPA_DEFNAME = 115	// server python identifier	Function or method name definition
const SCE_HPA_OPERATOR = 116	// server python operator	Operators
const SCE_HPA_IDENTIFIER = 117	// server python identifier	Identifiers
const SCE_HPHP_DEFAULT = 118	// server php default	Default
const SCE_HPHP_HSTRING = 119	// server php literal string	Double quoted String
const SCE_HPHP_SIMPLESTRING = 120	// server php literal string	Single quoted string
const SCE_HPHP_WORD = 121	// server php keyword	Keyword
const SCE_HPHP_NUMBER = 122	// server php literal numeric	Number
const SCE_HPHP_VARIABLE = 123	// server php identifier	Variable
const SCE_HPHP_COMMENT = 124	// server php comment	Comment
const SCE_HPHP_COMMENTLINE = 125	// server php comment line	One line comment
const SCE_HPHP_HSTRING_VARIABLE = 126	// server php literal string identifier	PHP variable in double quoted string
const SCE_HPHP_OPERATOR = 127	// server php operator	PHP operator

Please note that this lexer does HTML and XML with also embedded JavaScript, PHP or python.

James_Pitchford · February 2, 2022, 5:37pm

Thank you Christian.

So I understand that would be step 1 - that allows me to set all the colours for the various elements of html, xml, javascript etc.

But to do the other things like autocomplete, element collapsing etc, do I now need to identify all the keywords etc for each language?

Christian_Schmitz · February 2, 2022, 5:51pm

Yes, of course.
Check our example, which does already an auto complete for a list of keywords.
You would feed your list and may reuse the code.

Collapsing should work automatically.

James_Pitchford · February 2, 2022, 5:52pm

Thank you. Clearer now.

Can’t get collapsing to work yet but will investigate further.

Christian_Schmitz · February 4, 2022, 9:49am

I’ll make a html editor example.

Christian_Schmitz · February 4, 2022, 10:27am

and Windows

Christian_Schmitz · December 6, 2022, 9:50am

Already one year since we added this control.
Have you tried it?