Please file new bugs on Launchpad: Invirt or XVM (if you're not sure which, just pick one)

Context Navigation

source: trunk/packages/libyaml/src/scanner.c @ 1230

Last change on this file since 1230 was 898, checked in by hartmans, 16 years ago
Add pyyaml and libyaml packages backported from lenny. There is discussion about how these should go in the repository; these are added in this form in order to make forward progress.
File size: 95.8 KB

Line
1
2	/*
3	* Introduction
4	* ************
5	*
6	* The following notes assume that you are familiar with the YAML specification
7	* (http://yaml.org/spec/cvs/current.html). We mostly follow it, although in
8	* some cases we are less restrictive that it requires.
9	*
10	* The process of transforming a YAML stream into a sequence of events is
11	* divided on two steps: Scanning and Parsing.
12	*
13	* The Scanner transforms the input stream into a sequence of tokens, while the
14	* parser transform the sequence of tokens produced by the Scanner into a
15	* sequence of parsing events.
16	*
17	* The Scanner is rather clever and complicated. The Parser, on the contrary,
18	* is a straightforward implementation of a recursive-descendant parser (or,
19	* LL(1) parser, as it is usually called).
20	*
21	* Actually there are two issues of Scanning that might be called "clever", the
22	* rest is quite straightforward. The issues are "block collection start" and
23	* "simple keys". Both issues are explained below in details.
24	*
25	* Here the Scanning step is explained and implemented. We start with the list
26	* of all the tokens produced by the Scanner together with short descriptions.
27	*
28	* Now, tokens:
29	*
30	* STREAM-START(encoding) # The stream start.
31	* STREAM-END # The stream end.
32	* VERSION-DIRECTIVE(major,minor) # The '%YAML' directive.
33	* TAG-DIRECTIVE(handle,prefix) # The '%TAG' directive.
34	* DOCUMENT-START # '---'
35	* DOCUMENT-END # '...'
36	* BLOCK-SEQUENCE-START # Indentation increase denoting a block
37	* BLOCK-MAPPING-START # sequence or a block mapping.
38	* BLOCK-END # Indentation decrease.
39	* FLOW-SEQUENCE-START # '['
40	* FLOW-SEQUENCE-END # ']'
41	* BLOCK-SEQUENCE-START # '{'
42	* BLOCK-SEQUENCE-END # '}'
43	* BLOCK-ENTRY # '-'
44	* FLOW-ENTRY # ','
45	* KEY # '?' or nothing (simple keys).
46	* VALUE # ':'
47	* ALIAS(anchor) # '*anchor'
48	* ANCHOR(anchor) # '&anchor'
49	* TAG(handle,suffix) # '!handle!suffix'
50	* SCALAR(value,style) # A scalar.
51	*
52	* The following two tokens are "virtual" tokens denoting the beginning and the
53	* end of the stream:
54	*
55	* STREAM-START(encoding)
56	* STREAM-END
57	*
58	* We pass the information about the input stream encoding with the
59	* STREAM-START token.
60	*
61	* The next two tokens are responsible for tags:
62	*
63	* VERSION-DIRECTIVE(major,minor)
64	* TAG-DIRECTIVE(handle,prefix)
65	*
66	* Example:
67	*
68	* %YAML 1.1
69	* %TAG ! !foo
70	* %TAG !yaml! tag:yaml.org,2002:
71	* ---
72	*
73	* The correspoding sequence of tokens:
74	*
75	* STREAM-START(utf-8)
76	* VERSION-DIRECTIVE(1,1)
77	* TAG-DIRECTIVE("!","!foo")
78	* TAG-DIRECTIVE("!yaml","tag:yaml.org,2002:")
79	* DOCUMENT-START
80	* STREAM-END
81	*
82	* Note that the VERSION-DIRECTIVE and TAG-DIRECTIVE tokens occupy a whole
83	* line.
84	*
85	* The document start and end indicators are represented by:
86	*
87	* DOCUMENT-START
88	* DOCUMENT-END
89	*
90	* Note that if a YAML stream contains an implicit document (without '---'
91	* and '...' indicators), no DOCUMENT-START and DOCUMENT-END tokens will be
92	* produced.
93	*
94	* In the following examples, we present whole documents together with the
95	* produced tokens.
96	*
97	* 1. An implicit document:
98	*
99	* 'a scalar'
100	*
101	* Tokens:
102	*
103	* STREAM-START(utf-8)
104	* SCALAR("a scalar",single-quoted)
105	* STREAM-END
106	*
107	* 2. An explicit document:
108	*
109	* ---
110	* 'a scalar'
111	* ...
112	*
113	* Tokens:
114	*
115	* STREAM-START(utf-8)
116	* DOCUMENT-START
117	* SCALAR("a scalar",single-quoted)
118	* DOCUMENT-END
119	* STREAM-END
120	*
121	* 3. Several documents in a stream:
122	*
123	* 'a scalar'
124	* ---
125	* 'another scalar'
126	* ---
127	* 'yet another scalar'
128	*
129	* Tokens:
130	*
131	* STREAM-START(utf-8)
132	* SCALAR("a scalar",single-quoted)
133	* DOCUMENT-START
134	* SCALAR("another scalar",single-quoted)
135	* DOCUMENT-START
136	* SCALAR("yet another scalar",single-quoted)
137	* STREAM-END
138	*
139	* We have already introduced the SCALAR token above. The following tokens are
140	* used to describe aliases, anchors, tag, and scalars:
141	*
142	* ALIAS(anchor)
143	* ANCHOR(anchor)
144	* TAG(handle,suffix)
145	* SCALAR(value,style)
146	*
147	* The following series of examples illustrate the usage of these tokens:
148	*
149	* 1. A recursive sequence:
150	*
151	* &A [ *A ]
152	*
153	* Tokens:
154	*
155	* STREAM-START(utf-8)
156	* ANCHOR("A")
157	* FLOW-SEQUENCE-START
158	* ALIAS("A")
159	* FLOW-SEQUENCE-END
160	* STREAM-END
161	*
162	* 2. A tagged scalar:
163	*
164	* !!float "3.14" # A good approximation.
165	*
166	* Tokens:
167	*
168	* STREAM-START(utf-8)
169	* TAG("!!","float")
170	* SCALAR("3.14",double-quoted)
171	* STREAM-END
172	*
173	* 3. Various scalar styles:
174	*
175	* --- # Implicit empty plain scalars do not produce tokens.
176	* --- a plain scalar
177	* --- 'a single-quoted scalar'
178	* --- "a double-quoted scalar"
179	* --- \|-
180	* a literal scalar
181	* --- >-
182	* a folded
183	* scalar
184	*
185	* Tokens:
186	*
187	* STREAM-START(utf-8)
188	* DOCUMENT-START
189	* DOCUMENT-START
190	* SCALAR("a plain scalar",plain)
191	* DOCUMENT-START
192	* SCALAR("a single-quoted scalar",single-quoted)
193	* DOCUMENT-START
194	* SCALAR("a double-quoted scalar",double-quoted)
195	* DOCUMENT-START
196	* SCALAR("a literal scalar",literal)
197	* DOCUMENT-START
198	* SCALAR("a folded scalar",folded)
199	* STREAM-END
200	*
201	* Now it's time to review collection-related tokens. We will start with
202	* flow collections:
203	*
204	* FLOW-SEQUENCE-START
205	* FLOW-SEQUENCE-END
206	* FLOW-MAPPING-START
207	* FLOW-MAPPING-END
208	* FLOW-ENTRY
209	* KEY
210	* VALUE
211	*
212	* The tokens FLOW-SEQUENCE-START, FLOW-SEQUENCE-END, FLOW-MAPPING-START, and
213	* FLOW-MAPPING-END represent the indicators '[', ']', '{', and '}'
214	* correspondingly. FLOW-ENTRY represent the ',' indicator. Finally the
215	* indicators '?' and ':', which are used for denoting mapping keys and values,
216	* are represented by the KEY and VALUE tokens.
217	*
218	* The following examples show flow collections:
219	*
220	* 1. A flow sequence:
221	*
222	* [item 1, item 2, item 3]
223	*
224	* Tokens:
225	*
226	* STREAM-START(utf-8)
227	* FLOW-SEQUENCE-START
228	* SCALAR("item 1",plain)
229	* FLOW-ENTRY
230	* SCALAR("item 2",plain)
231	* FLOW-ENTRY
232	* SCALAR("item 3",plain)
233	* FLOW-SEQUENCE-END
234	* STREAM-END
235	*
236	* 2. A flow mapping:
237	*
238	* {
239	* a simple key: a value, # Note that the KEY token is produced.
240	* ? a complex key: another value,
241	* }
242	*
243	* Tokens:
244	*
245	* STREAM-START(utf-8)
246	* FLOW-MAPPING-START
247	* KEY
248	* SCALAR("a simple key",plain)
249	* VALUE
250	* SCALAR("a value",plain)
251	* FLOW-ENTRY
252	* KEY
253	* SCALAR("a complex key",plain)
254	* VALUE
255	* SCALAR("another value",plain)
256	* FLOW-ENTRY
257	* FLOW-MAPPING-END
258	* STREAM-END
259	*
260	* A simple key is a key which is not denoted by the '?' indicator. Note that
261	* the Scanner still produce the KEY token whenever it encounters a simple key.
262	*
263	* For scanning block collections, the following tokens are used (note that we
264	* repeat KEY and VALUE here):
265	*
266	* BLOCK-SEQUENCE-START
267	* BLOCK-MAPPING-START
268	* BLOCK-END
269	* BLOCK-ENTRY
270	* KEY
271	* VALUE
272	*
273	* The tokens BLOCK-SEQUENCE-START and BLOCK-MAPPING-START denote indentation
274	* increase that precedes a block collection (cf. the INDENT token in Python).
275	* The token BLOCK-END denote indentation decrease that ends a block collection
276	* (cf. the DEDENT token in Python). However YAML has some syntax pecularities
277	* that makes detections of these tokens more complex.
278	*
279	* The tokens BLOCK-ENTRY, KEY, and VALUE are used to represent the indicators
280	* '-', '?', and ':' correspondingly.
281	*
282	* The following examples show how the tokens BLOCK-SEQUENCE-START,
283	* BLOCK-MAPPING-START, and BLOCK-END are emitted by the Scanner:
284	*
285	* 1. Block sequences:
286	*
287	* - item 1
288	* - item 2
289	* -
290	* - item 3.1
291	* - item 3.2
292	* -
293	* key 1: value 1
294	* key 2: value 2
295	*
296	* Tokens:
297	*
298	* STREAM-START(utf-8)
299	* BLOCK-SEQUENCE-START
300	* BLOCK-ENTRY
301	* SCALAR("item 1",plain)
302	* BLOCK-ENTRY
303	* SCALAR("item 2",plain)
304	* BLOCK-ENTRY
305	* BLOCK-SEQUENCE-START
306	* BLOCK-ENTRY
307	* SCALAR("item 3.1",plain)
308	* BLOCK-ENTRY
309	* SCALAR("item 3.2",plain)
310	* BLOCK-END
311	* BLOCK-ENTRY
312	* BLOCK-MAPPING-START
313	* KEY
314	* SCALAR("key 1",plain)
315	* VALUE
316	* SCALAR("value 1",plain)
317	* KEY
318	* SCALAR("key 2",plain)
319	* VALUE
320	* SCALAR("value 2",plain)
321	* BLOCK-END
322	* BLOCK-END
323	* STREAM-END
324	*
325	* 2. Block mappings:
326	*
327	* a simple key: a value # The KEY token is produced here.
328	* ? a complex key
329	* : another value
330	* a mapping:
331	* key 1: value 1
332	* key 2: value 2
333	* a sequence:
334	* - item 1
335	* - item 2
336	*
337	* Tokens:
338	*
339	* STREAM-START(utf-8)
340	* BLOCK-MAPPING-START
341	* KEY
342	* SCALAR("a simple key",plain)
343	* VALUE
344	* SCALAR("a value",plain)
345	* KEY
346	* SCALAR("a complex key",plain)
347	* VALUE
348	* SCALAR("another value",plain)
349	* KEY
350	* SCALAR("a mapping",plain)
351	* BLOCK-MAPPING-START
352	* KEY
353	* SCALAR("key 1",plain)
354	* VALUE
355	* SCALAR("value 1",plain)
356	* KEY
357	* SCALAR("key 2",plain)
358	* VALUE
359	* SCALAR("value 2",plain)
360	* BLOCK-END
361	* KEY
362	* SCALAR("a sequence",plain)
363	* VALUE
364	* BLOCK-SEQUENCE-START
365	* BLOCK-ENTRY
366	* SCALAR("item 1",plain)
367	* BLOCK-ENTRY
368	* SCALAR("item 2",plain)
369	* BLOCK-END
370	* BLOCK-END
371	* STREAM-END
372	*
373	* YAML does not always require to start a new block collection from a new
374	* line. If the current line contains only '-', '?', and ':' indicators, a new
375	* block collection may start at the current line. The following examples
376	* illustrate this case:
377	*
378	* 1. Collections in a sequence:
379	*
380	* - - item 1
381	* - item 2
382	* - key 1: value 1
383	* key 2: value 2
384	* - ? complex key
385	* : complex value
386	*
387	* Tokens:
388	*
389	* STREAM-START(utf-8)
390	* BLOCK-SEQUENCE-START
391	* BLOCK-ENTRY
392	* BLOCK-SEQUENCE-START
393	* BLOCK-ENTRY
394	* SCALAR("item 1",plain)
395	* BLOCK-ENTRY
396	* SCALAR("item 2",plain)
397	* BLOCK-END
398	* BLOCK-ENTRY
399	* BLOCK-MAPPING-START
400	* KEY
401	* SCALAR("key 1",plain)
402	* VALUE
403	* SCALAR("value 1",plain)
404	* KEY
405	* SCALAR("key 2",plain)
406	* VALUE
407	* SCALAR("value 2",plain)
408	* BLOCK-END
409	* BLOCK-ENTRY
410	* BLOCK-MAPPING-START
411	* KEY
412	* SCALAR("complex key")
413	* VALUE
414	* SCALAR("complex value")
415	* BLOCK-END
416	* BLOCK-END
417	* STREAM-END
418	*
419	* 2. Collections in a mapping:
420	*
421	* ? a sequence
422	* : - item 1
423	* - item 2
424	* ? a mapping
425	* : key 1: value 1
426	* key 2: value 2
427	*
428	* Tokens:
429	*
430	* STREAM-START(utf-8)
431	* BLOCK-MAPPING-START
432	* KEY
433	* SCALAR("a sequence",plain)
434	* VALUE
435	* BLOCK-SEQUENCE-START
436	* BLOCK-ENTRY
437	* SCALAR("item 1",plain)
438	* BLOCK-ENTRY
439	* SCALAR("item 2",plain)
440	* BLOCK-END
441	* KEY
442	* SCALAR("a mapping",plain)
443	* VALUE
444	* BLOCK-MAPPING-START
445	* KEY
446	* SCALAR("key 1",plain)
447	* VALUE
448	* SCALAR("value 1",plain)
449	* KEY
450	* SCALAR("key 2",plain)
451	* VALUE
452	* SCALAR("value 2",plain)
453	* BLOCK-END
454	* BLOCK-END
455	* STREAM-END
456	*
457	* YAML also permits non-indented sequences if they are included into a block
458	* mapping. In this case, the token BLOCK-SEQUENCE-START is not produced:
459	*
460	* key:
461	* - item 1 # BLOCK-SEQUENCE-START is NOT produced here.
462	* - item 2
463	*
464	* Tokens:
465	*
466	* STREAM-START(utf-8)
467	* BLOCK-MAPPING-START
468	* KEY
469	* SCALAR("key",plain)
470	* VALUE
471	* BLOCK-ENTRY
472	* SCALAR("item 1",plain)
473	* BLOCK-ENTRY
474	* SCALAR("item 2",plain)
475	* BLOCK-END
476	*/
477
478	#include "yaml_private.h"
479
480	/*
481	* Ensure that the buffer contains the required number of characters.
482	* Return 1 on success, 0 on failure (reader error or memory error).
483	*/
484
485	#define CACHE(parser,length) \
486	(parser->unread >= (length) \
487	? 1 \
488	: yaml_parser_update_buffer(parser, (length)))
489
490	/*
491	* Advance the buffer pointer.
492	*/
493
494	#define SKIP(parser) \
495	(parser->mark.index ++, \
496	parser->mark.column ++, \
497	parser->unread --, \
498	parser->buffer.pointer += WIDTH(parser->buffer))
499
500	#define SKIP_LINE(parser) \
501	(IS_CRLF(parser->buffer) ? \
502	(parser->mark.index += 2, \
503	parser->mark.column = 0, \
504	parser->mark.line ++, \
505	parser->unread -= 2, \
506	parser->buffer.pointer += 2) : \
507	IS_BREAK(parser->buffer) ? \
508	(parser->mark.index ++, \
509	parser->mark.column = 0, \
510	parser->mark.line ++, \
511	parser->unread --, \
512	parser->buffer.pointer += WIDTH(parser->buffer)) : 0)
513
514	/*
515	* Copy a character to a string buffer and advance pointers.
516	*/
517
518	#define READ(parser,string) \
519	(STRING_EXTEND(parser,string) ? \
520	(COPY(string,parser->buffer), \
521	parser->mark.index ++, \
522	parser->mark.column ++, \
523	parser->unread --, \
524	1) : 0)
525
526	/*
527	* Copy a line break character to a string buffer and advance pointers.
528	*/
529
530	#define READ_LINE(parser,string) \
531	(STRING_EXTEND(parser,string) ? \
532	(((CHECK_AT(parser->buffer,'\r',0) \
533	&& CHECK_AT(parser->buffer,'\n',1)) ? /* CR LF -> LF */ \
534	(*((string).pointer++) = (yaml_char_t) '\n', \
535	parser->buffer.pointer += 2, \
536	parser->mark.index += 2, \
537	parser->mark.column = 0, \
538	parser->mark.line ++, \
539	parser->unread -= 2) : \
540	(CHECK_AT(parser->buffer,'\r',0) \
541	\|\| CHECK_AT(parser->buffer,'\n',0)) ? /* CR\|LF -> LF */ \
542	(*((string).pointer++) = (yaml_char_t) '\n', \
543	parser->buffer.pointer ++, \
544	parser->mark.index ++, \
545	parser->mark.column = 0, \
546	parser->mark.line ++, \
547	parser->unread --) : \
548	(CHECK_AT(parser->buffer,'\xC2',0) \
549	&& CHECK_AT(parser->buffer,'\x85',1)) ? /* NEL -> LF */ \
550	(*((string).pointer++) = (yaml_char_t) '\n', \
551	parser->buffer.pointer += 2, \
552	parser->mark.index ++, \
553	parser->mark.column = 0, \
554	parser->mark.line ++, \
555	parser->unread --) : \
556	(CHECK_AT(parser->buffer,'\xE2',0) && \
557	CHECK_AT(parser->buffer,'\x80',1) && \
558	(CHECK_AT(parser->buffer,'\xA8',2) \|\| \
559	CHECK_AT(parser->buffer,'\xA9',2))) ? /* LS\|PS -> LS\|PS */ \
560	(((string).pointer++) = (parser->buffer.pointer++), \
561	((string).pointer++) = (parser->buffer.pointer++), \
562	((string).pointer++) = (parser->buffer.pointer++), \
563	parser->mark.index ++, \
564	parser->mark.column = 0, \
565	parser->mark.line ++, \
566	parser->unread --) : 0), \
567	1) : 0)
568
569	/*
570	* Public API declarations.
571	*/
572
573	YAML_DECLARE(int)
574	yaml_parser_scan(yaml_parser_t parser, yaml_token_t token);
575
576	/*
577	* Error handling.
578	*/
579
580	static int
581	yaml_parser_set_scanner_error(yaml_parser_t parser, const char context,
582	yaml_mark_t context_mark, const char *problem);
583
584	/*
585	* High-level token API.
586	*/
587
588	YAML_DECLARE(int)
589	yaml_parser_fetch_more_tokens(yaml_parser_t *parser);
590
591	static int
592	yaml_parser_fetch_next_token(yaml_parser_t *parser);
593
594	/*
595	* Potential simple keys.
596	*/
597
598	static int
599	yaml_parser_stale_simple_keys(yaml_parser_t *parser);
600
601	static int
602	yaml_parser_save_simple_key(yaml_parser_t *parser);
603
604	static int
605	yaml_parser_remove_simple_key(yaml_parser_t *parser);
606
607	static int
608	yaml_parser_increase_flow_level(yaml_parser_t *parser);
609
610	static int
611	yaml_parser_decrease_flow_level(yaml_parser_t *parser);
612
613	/*
614	* Indentation treatment.
615	*/
616
617	static int
618	yaml_parser_roll_indent(yaml_parser_t *parser, int column,
619	int number, yaml_token_type_t type, yaml_mark_t mark);
620
621	static int
622	yaml_parser_unroll_indent(yaml_parser_t *parser, int column);
623
624	/*
625	* Token fetchers.
626	*/
627
628	static int
629	yaml_parser_fetch_stream_start(yaml_parser_t *parser);
630
631	static int
632	yaml_parser_fetch_stream_end(yaml_parser_t *parser);
633
634	static int
635	yaml_parser_fetch_directive(yaml_parser_t *parser);
636
637	static int
638	yaml_parser_fetch_document_indicator(yaml_parser_t *parser,
639	yaml_token_type_t type);
640
641	static int
642	yaml_parser_fetch_flow_collection_start(yaml_parser_t *parser,
643	yaml_token_type_t type);
644
645	static int
646	yaml_parser_fetch_flow_collection_end(yaml_parser_t *parser,
647	yaml_token_type_t type);
648
649	static int
650	yaml_parser_fetch_flow_entry(yaml_parser_t *parser);
651
652	static int
653	yaml_parser_fetch_block_entry(yaml_parser_t *parser);
654
655	static int
656	yaml_parser_fetch_key(yaml_parser_t *parser);
657
658	static int
659	yaml_parser_fetch_value(yaml_parser_t *parser);
660
661	static int
662	yaml_parser_fetch_anchor(yaml_parser_t *parser, yaml_token_type_t type);
663
664	static int
665	yaml_parser_fetch_tag(yaml_parser_t *parser);
666
667	static int
668	yaml_parser_fetch_block_scalar(yaml_parser_t *parser, int literal);
669
670	static int
671	yaml_parser_fetch_flow_scalar(yaml_parser_t *parser, int single);
672
673	static int
674	yaml_parser_fetch_plain_scalar(yaml_parser_t *parser);
675
676	/*
677	* Token scanners.
678	*/
679
680	static int
681	yaml_parser_scan_to_next_token(yaml_parser_t *parser);
682
683	static int
684	yaml_parser_scan_directive(yaml_parser_t parser, yaml_token_t token);
685
686	static int
687	yaml_parser_scan_directive_name(yaml_parser_t *parser,
688	yaml_mark_t start_mark, yaml_char_t **name);
689
690	static int
691	yaml_parser_scan_version_directive_value(yaml_parser_t *parser,
692	yaml_mark_t start_mark, int major, int minor);
693
694	static int
695	yaml_parser_scan_version_directive_number(yaml_parser_t *parser,
696	yaml_mark_t start_mark, int *number);
697
698	static int
699	yaml_parser_scan_tag_directive_value(yaml_parser_t *parser,
700	yaml_mark_t mark, yaml_char_t handle, yaml_char_t prefix);
701
702	static int
703	yaml_parser_scan_anchor(yaml_parser_t parser, yaml_token_t token,
704	yaml_token_type_t type);
705
706	static int
707	yaml_parser_scan_tag(yaml_parser_t parser, yaml_token_t token);
708
709	static int
710	yaml_parser_scan_tag_handle(yaml_parser_t *parser, int directive,
711	yaml_mark_t start_mark, yaml_char_t **handle);
712
713	static int
714	yaml_parser_scan_tag_uri(yaml_parser_t *parser, int directive,
715	yaml_char_t head, yaml_mark_t start_mark, yaml_char_t *uri);
716
717	static int
718	yaml_parser_scan_uri_escapes(yaml_parser_t *parser, int directive,
719	yaml_mark_t start_mark, yaml_string_t *string);
720
721	static int
722	yaml_parser_scan_block_scalar(yaml_parser_t parser, yaml_token_t token,
723	int literal);
724
725	static int
726	yaml_parser_scan_block_scalar_breaks(yaml_parser_t *parser,
727	int indent, yaml_string_t breaks,
728	yaml_mark_t start_mark, yaml_mark_t *end_mark);
729
730	static int
731	yaml_parser_scan_flow_scalar(yaml_parser_t parser, yaml_token_t token,
732	int single);
733
734	static int
735	yaml_parser_scan_plain_scalar(yaml_parser_t parser, yaml_token_t token);
736
737	/*
738	* Get the next token.
739	*/
740
741	YAML_DECLARE(int)
742	yaml_parser_scan(yaml_parser_t parser, yaml_token_t token)
743	{
744	assert(parser); /* Non-NULL parser object is expected. */
745	assert(token); /* Non-NULL token object is expected. */
746
747	/* Erase the token object. */
748
749	memset(token, 0, sizeof(yaml_token_t));
750
751	/* No tokens after STREAM-END or error. */
752
753	if (parser->stream_end_produced \|\| parser->error) {
754	return 1;
755	}
756
757	/* Ensure that the tokens queue contains enough tokens. */
758
759	if (!parser->token_available) {
760	if (!yaml_parser_fetch_more_tokens(parser))
761	return 0;
762	}
763
764	/* Fetch the next token from the queue. */
765
766	*token = DEQUEUE(parser, parser->tokens);
767	parser->token_available = 0;
768	parser->tokens_parsed ++;
769
770	if (token->type == YAML_STREAM_END_TOKEN) {
771	parser->stream_end_produced = 1;
772	}
773
774	return 1;
775	}
776
777	/*
778	* Set the scanner error and return 0.
779	*/
780
781	static int
782	yaml_parser_set_scanner_error(yaml_parser_t parser, const char context,
783	yaml_mark_t context_mark, const char *problem)
784	{
785	parser->error = YAML_SCANNER_ERROR;
786	parser->context = context;
787	parser->context_mark = context_mark;
788	parser->problem = problem;
789	parser->problem_mark = parser->mark;
790
791	return 0;
792	}
793
794	/*
795	* Ensure that the tokens queue contains at least one token which can be
796	* returned to the Parser.
797	*/
798
799	YAML_DECLARE(int)
800	yaml_parser_fetch_more_tokens(yaml_parser_t *parser)
801	{
802	int need_more_tokens;
803
804	/* While we need more tokens to fetch, do it. */
805
806	while (1)
807	{
808	/*
809	* Check if we really need to fetch more tokens.
810	*/
811
812	need_more_tokens = 0;
813
814	if (parser->tokens.head == parser->tokens.tail)
815	{
816	/* Queue is empty. */
817
818	need_more_tokens = 1;
819	}
820	else
821	{
822	yaml_simple_key_t *simple_key;
823
824	/* Check if any potential simple key may occupy the head position. */
825
826	if (!yaml_parser_stale_simple_keys(parser))
827	return 0;
828
829	for (simple_key = parser->simple_keys.start;
830	simple_key != parser->simple_keys.top; simple_key++) {
831	if (simple_key->possible
832	&& simple_key->token_number == parser->tokens_parsed) {
833	need_more_tokens = 1;
834	break;
835	}
836	}
837	}
838
839	/* We are finished. */
840
841	if (!need_more_tokens)
842	break;
843
844	/* Fetch the next token. */
845
846	if (!yaml_parser_fetch_next_token(parser))
847	return 0;
848	}
849
850	parser->token_available = 1;
851
852	return 1;
853	}
854
855	/*
856	* The dispatcher for token fetchers.
857	*/
858
859	static int
860	yaml_parser_fetch_next_token(yaml_parser_t *parser)
861	{
862	/* Ensure that the buffer is initialized. */
863
864	if (!CACHE(parser, 1))
865	return 0;
866
867	/* Check if we just started scanning. Fetch STREAM-START then. */
868
869	if (!parser->stream_start_produced)
870	return yaml_parser_fetch_stream_start(parser);
871
872	/* Eat whitespaces and comments until we reach the next token. */
873
874	if (!yaml_parser_scan_to_next_token(parser))
875	return 0;
876
877	/* Remove obsolete potential simple keys. */
878
879	if (!yaml_parser_stale_simple_keys(parser))
880	return 0;
881
882	/* Check the indentation level against the current column. */
883
884	if (!yaml_parser_unroll_indent(parser, parser->mark.column))
885	return 0;
886
887	/*
888	* Ensure that the buffer contains at least 4 characters. 4 is the length
889	* of the longest indicators ('--- ' and '... ').
890	*/
891
892	if (!CACHE(parser, 4))
893	return 0;
894
895	/* Is it the end of the stream? */
896
897	if (IS_Z(parser->buffer))
898	return yaml_parser_fetch_stream_end(parser);
899
900	/* Is it a directive? */
901
902	if (parser->mark.column == 0 && CHECK(parser->buffer, '%'))
903	return yaml_parser_fetch_directive(parser);
904
905	/* Is it the document start indicator? */
906
907	if (parser->mark.column == 0
908	&& CHECK_AT(parser->buffer, '-', 0)
909	&& CHECK_AT(parser->buffer, '-', 1)
910	&& CHECK_AT(parser->buffer, '-', 2)
911	&& IS_BLANKZ_AT(parser->buffer, 3))
912	return yaml_parser_fetch_document_indicator(parser,
913	YAML_DOCUMENT_START_TOKEN);
914
915	/* Is it the document end indicator? */
916
917	if (parser->mark.column == 0
918	&& CHECK_AT(parser->buffer, '.', 0)
919	&& CHECK_AT(parser->buffer, '.', 1)
920	&& CHECK_AT(parser->buffer, '.', 2)
921	&& IS_BLANKZ_AT(parser->buffer, 3))
922	return yaml_parser_fetch_document_indicator(parser,
923	YAML_DOCUMENT_END_TOKEN);
924
925	/* Is it the flow sequence start indicator? */
926
927	if (CHECK(parser->buffer, '['))
928	return yaml_parser_fetch_flow_collection_start(parser,
929	YAML_FLOW_SEQUENCE_START_TOKEN);
930
931	/* Is it the flow mapping start indicator? */
932
933	if (CHECK(parser->buffer, '{'))
934	return yaml_parser_fetch_flow_collection_start(parser,
935	YAML_FLOW_MAPPING_START_TOKEN);
936
937	/* Is it the flow sequence end indicator? */
938
939	if (CHECK(parser->buffer, ']'))
940	return yaml_parser_fetch_flow_collection_end(parser,
941	YAML_FLOW_SEQUENCE_END_TOKEN);
942
943	/* Is it the flow mapping end indicator? */
944
945	if (CHECK(parser->buffer, '}'))
946	return yaml_parser_fetch_flow_collection_end(parser,
947	YAML_FLOW_MAPPING_END_TOKEN);
948
949	/* Is it the flow entry indicator? */
950
951	if (CHECK(parser->buffer, ','))
952	return yaml_parser_fetch_flow_entry(parser);
953
954	/* Is it the block entry indicator? */
955
956	if (CHECK(parser->buffer, '-') && IS_BLANKZ_AT(parser->buffer, 1))
957	return yaml_parser_fetch_block_entry(parser);
958
959	/* Is it the key indicator? */
960
961	if (CHECK(parser->buffer, '?')
962	&& (parser->flow_level \|\| IS_BLANKZ_AT(parser->buffer, 1)))
963	return yaml_parser_fetch_key(parser);
964
965	/* Is it the value indicator? */
966
967	if (CHECK(parser->buffer, ':')
968	&& (parser->flow_level \|\| IS_BLANKZ_AT(parser->buffer, 1)))
969	return yaml_parser_fetch_value(parser);
970
971	/* Is it an alias? */
972
973	if (CHECK(parser->buffer, '*'))
974	return yaml_parser_fetch_anchor(parser, YAML_ALIAS_TOKEN);
975
976	/* Is it an anchor? */
977
978	if (CHECK(parser->buffer, '&'))
979	return yaml_parser_fetch_anchor(parser, YAML_ANCHOR_TOKEN);
980
981	/* Is it a tag? */
982
983	if (CHECK(parser->buffer, '!'))
984	return yaml_parser_fetch_tag(parser);
985
986	/* Is it a literal scalar? */
987
988	if (CHECK(parser->buffer, '\|') && !parser->flow_level)
989	return yaml_parser_fetch_block_scalar(parser, 1);
990
991	/* Is it a folded scalar? */
992
993	if (CHECK(parser->buffer, '>') && !parser->flow_level)
994	return yaml_parser_fetch_block_scalar(parser, 0);
995
996	/* Is it a single-quoted scalar? */
997
998	if (CHECK(parser->buffer, '\''))
999	return yaml_parser_fetch_flow_scalar(parser, 1);
1000
1001	/* Is it a double-quoted scalar? */
1002
1003	if (CHECK(parser->buffer, '"'))
1004	return yaml_parser_fetch_flow_scalar(parser, 0);
1005
1006	/*
1007	* Is it a plain scalar?
1008	*
1009	* A plain scalar may start with any non-blank characters except
1010	*
1011	* '-', '?', ':', ',', '[', ']', '{', '}',
1012	* '#', '&', '*', '!', '\|', '>', '\'', '\"',
1013	* '%', '@', '`'.
1014	*
1015	* In the block context (and, for the '-' indicator, in the flow context
1016	* too), it may also start with the characters
1017	*
1018	* '-', '?', ':'
1019	*
1020	* if it is followed by a non-space character.
1021	*
1022	* The last rule is more restrictive than the specification requires.
1023	*/
1024
1025	if (!(IS_BLANKZ(parser->buffer) \|\| CHECK(parser->buffer, '-')
1026	\|\| CHECK(parser->buffer, '?') \|\| CHECK(parser->buffer, ':')
1027	\|\| CHECK(parser->buffer, ',') \|\| CHECK(parser->buffer, '[')
1028	\|\| CHECK(parser->buffer, ']') \|\| CHECK(parser->buffer, '{')
1029	\|\| CHECK(parser->buffer, '}') \|\| CHECK(parser->buffer, '#')
1030	\|\| CHECK(parser->buffer, '&') \|\| CHECK(parser->buffer, '*')
1031	\|\| CHECK(parser->buffer, '!') \|\| CHECK(parser->buffer, '\|')
1032	\|\| CHECK(parser->buffer, '>') \|\| CHECK(parser->buffer, '\'')
1033	\|\| CHECK(parser->buffer, '"') \|\| CHECK(parser->buffer, '%')
1034	\|\| CHECK(parser->buffer, '@') \|\| CHECK(parser->buffer, '`')) \|\|
1035	(CHECK(parser->buffer, '-') && !IS_BLANK_AT(parser->buffer, 1)) \|\|
1036	(!parser->flow_level &&
1037	(CHECK(parser->buffer, '?') \|\| CHECK(parser->buffer, ':'))
1038	&& !IS_BLANKZ_AT(parser->buffer, 1)))
1039	return yaml_parser_fetch_plain_scalar(parser);
1040
1041	/*
1042	* If we don't determine the token type so far, it is an error.
1043	*/
1044
1045	return yaml_parser_set_scanner_error(parser,
1046	"while scanning for the next token", parser->mark,
1047	"found character that cannot start any token");
1048	}
1049
1050	/*
1051	* Check the list of potential simple keys and remove the positions that
1052	* cannot contain simple keys anymore.
1053	*/
1054
1055	static int
1056	yaml_parser_stale_simple_keys(yaml_parser_t *parser)
1057	{
1058	yaml_simple_key_t *simple_key;
1059
1060	/* Check for a potential simple key for each flow level. */
1061
1062	for (simple_key = parser->simple_keys.start;
1063	simple_key != parser->simple_keys.top; simple_key ++)
1064	{
1065	/*
1066	* The specification requires that a simple key
1067	*
1068	* - is limited to a single line,
1069	* - is shorter than 1024 characters.
1070	*/
1071
1072	if (simple_key->possible
1073	&& (simple_key->mark.line < parser->mark.line
1074	\|\| simple_key->mark.index+1024 < parser->mark.index)) {
1075
1076	/* Check if the potential simple key to be removed is required. */
1077
1078	if (simple_key->required) {
1079	return yaml_parser_set_scanner_error(parser,
1080	"while scanning a simple key", simple_key->mark,
1081	"could not found expected ':'");
1082	}
1083
1084	simple_key->possible = 0;
1085	}
1086	}
1087
1088	return 1;
1089	}
1090
1091	/*
1092	* Check if a simple key may start at the current position and add it if
1093	* needed.
1094	*/
1095
1096	static int
1097	yaml_parser_save_simple_key(yaml_parser_t *parser)
1098	{
1099	/*
1100	* A simple key is required at the current position if the scanner is in
1101	* the block context and the current column coincides with the indentation
1102	* level.
1103	*/
1104
1105	int required = (!parser->flow_level
1106	&& parser->indent == (int)parser->mark.column);
1107
1108	/*
1109	* A simple key is required only when it is the first token in the current
1110	* line. Therefore it is always allowed. But we add a check anyway.
1111	*/
1112
1113	assert(parser->simple_key_allowed \|\| !required); /* Impossible. */
1114
1115	/*
1116	* If the current position may start a simple key, save it.
1117	*/
1118
1119	if (parser->simple_key_allowed)
1120	{
1121	yaml_simple_key_t simple_key = { 1, required,
1122	parser->tokens_parsed + parser->tokens.tail - parser->tokens.head,
1123	{ 0, 0, 0 } };
1124	simple_key.mark = parser->mark;
1125
1126	if (!yaml_parser_remove_simple_key(parser)) return 0;
1127
1128	*(parser->simple_keys.top-1) = simple_key;
1129	}
1130
1131	return 1;
1132	}
1133
1134	/*
1135	* Remove a potential simple key at the current flow level.
1136	*/
1137
1138	static int
1139	yaml_parser_remove_simple_key(yaml_parser_t *parser)
1140	{
1141	yaml_simple_key_t *simple_key = parser->simple_keys.top-1;
1142
1143	if (simple_key->possible)
1144	{
1145	/* If the key is required, it is an error. */
1146
1147	if (simple_key->required) {
1148	return yaml_parser_set_scanner_error(parser,
1149	"while scanning a simple key", simple_key->mark,
1150	"could not found expected ':'");
1151	}
1152	}
1153
1154	/* Remove the key from the stack. */
1155
1156	simple_key->possible = 0;
1157
1158	return 1;
1159	}
1160
1161	/*
1162	* Increase the flow level and resize the simple key list if needed.
1163	*/
1164
1165	static int
1166	yaml_parser_increase_flow_level(yaml_parser_t *parser)
1167	{
1168	yaml_simple_key_t empty_simple_key = { 0, 0, 0, { 0, 0, 0 } };
1169
1170	/* Reset the simple key on the next level. */
1171
1172	if (!PUSH(parser, parser->simple_keys, empty_simple_key))
1173	return 0;
1174
1175	/* Increase the flow level. */
1176
1177	parser->flow_level++;
1178
1179	return 1;
1180	}
1181
1182	/*
1183	* Decrease the flow level.
1184	*/
1185
1186	static int
1187	yaml_parser_decrease_flow_level(yaml_parser_t *parser)
1188	{
1189	yaml_simple_key_t dummy_key; /* Used to eliminate a compiler warning. */
1190
1191	if (parser->flow_level) {
1192	parser->flow_level --;
1193	dummy_key = POP(parser, parser->simple_keys);
1194	}
1195
1196	return 1;
1197	}
1198
1199	/*
1200	* Push the current indentation level to the stack and set the new level
1201	* the current column is greater than the indentation level. In this case,
1202	* append or insert the specified token into the token queue.
1203	*
1204	*/
1205
1206	static int
1207	yaml_parser_roll_indent(yaml_parser_t *parser, int column,
1208	int number, yaml_token_type_t type, yaml_mark_t mark)
1209	{
1210	yaml_token_t token;
1211
1212	/* In the flow context, do nothing. */
1213
1214	if (parser->flow_level)
1215	return 1;
1216
1217	if (parser->indent < column)
1218	{
1219	/*
1220	* Push the current indentation level to the stack and set the new
1221	* indentation level.
1222	*/
1223
1224	if (!PUSH(parser, parser->indents, parser->indent))
1225	return 0;
1226
1227	parser->indent = column;
1228
1229	/* Create a token and insert it into the queue. */
1230
1231	TOKEN_INIT(token, type, mark, mark);
1232
1233	if (number == -1) {
1234	if (!ENQUEUE(parser, parser->tokens, token))
1235	return 0;
1236	}
1237	else {
1238	if (!QUEUE_INSERT(parser,
1239	parser->tokens, number - parser->tokens_parsed, token))
1240	return 0;
1241	}
1242	}
1243
1244	return 1;
1245	}
1246
1247	/*
1248	* Pop indentation levels from the indents stack until the current level
1249	* becomes less or equal to the column. For each intendation level, append
1250	* the BLOCK-END token.
1251	*/
1252
1253
1254	static int
1255	yaml_parser_unroll_indent(yaml_parser_t *parser, int column)
1256	{
1257	yaml_token_t token;
1258
1259	/* In the flow context, do nothing. */
1260
1261	if (parser->flow_level)
1262	return 1;
1263
1264	/* Loop through the intendation levels in the stack. */
1265
1266	while (parser->indent > column)
1267	{
1268	/* Create a token and append it to the queue. */
1269
1270	TOKEN_INIT(token, YAML_BLOCK_END_TOKEN, parser->mark, parser->mark);
1271
1272	if (!ENQUEUE(parser, parser->tokens, token))
1273	return 0;
1274
1275	/* Pop the indentation level. */
1276
1277	parser->indent = POP(parser, parser->indents);
1278	}
1279
1280	return 1;
1281	}
1282
1283	/*
1284	* Initialize the scanner and produce the STREAM-START token.
1285	*/
1286
1287	static int
1288	yaml_parser_fetch_stream_start(yaml_parser_t *parser)
1289	{
1290	yaml_simple_key_t simple_key = { 0, 0, 0, { 0, 0, 0 } };
1291	yaml_token_t token;
1292
1293	/* Set the initial indentation. */
1294
1295	parser->indent = -1;
1296
1297	/* Initialize the simple key stack. */
1298
1299	if (!PUSH(parser, parser->simple_keys, simple_key))
1300	return 0;
1301
1302	/* A simple key is allowed at the beginning of the stream. */
1303
1304	parser->simple_key_allowed = 1;
1305
1306	/* We have started. */
1307
1308	parser->stream_start_produced = 1;
1309
1310	/* Create the STREAM-START token and append it to the queue. */
1311
1312	STREAM_START_TOKEN_INIT(token, parser->encoding,
1313	parser->mark, parser->mark);
1314
1315	if (!ENQUEUE(parser, parser->tokens, token))
1316	return 0;
1317
1318	return 1;
1319	}
1320
1321	/*
1322	* Produce the STREAM-END token and shut down the scanner.
1323	*/
1324
1325	static int
1326	yaml_parser_fetch_stream_end(yaml_parser_t *parser)
1327	{
1328	yaml_token_t token;
1329
1330	/* Force new line. */
1331
1332	if (parser->mark.column != 0) {
1333	parser->mark.column = 0;
1334	parser->mark.line ++;
1335	}
1336
1337	/* Reset the indentation level. */
1338
1339	if (!yaml_parser_unroll_indent(parser, -1))
1340	return 0;
1341
1342	/* Reset simple keys. */
1343
1344	if (!yaml_parser_remove_simple_key(parser))
1345	return 0;
1346
1347	parser->simple_key_allowed = 0;
1348
1349	/* Create the STREAM-END token and append it to the queue. */
1350
1351	STREAM_END_TOKEN_INIT(token, parser->mark, parser->mark);
1352
1353	if (!ENQUEUE(parser, parser->tokens, token))
1354	return 0;
1355
1356	return 1;
1357	}
1358
1359	/*
1360	* Produce a VERSION-DIRECTIVE or TAG-DIRECTIVE token.
1361	*/
1362
1363	static int
1364	yaml_parser_fetch_directive(yaml_parser_t *parser)
1365	{
1366	yaml_token_t token;
1367
1368	/* Reset the indentation level. */
1369
1370	if (!yaml_parser_unroll_indent(parser, -1))
1371	return 0;
1372
1373	/* Reset simple keys. */
1374
1375	if (!yaml_parser_remove_simple_key(parser))
1376	return 0;
1377
1378	parser->simple_key_allowed = 0;
1379
1380	/* Create the YAML-DIRECTIVE or TAG-DIRECTIVE token. */
1381
1382	if (!yaml_parser_scan_directive(parser, &token))
1383	return 0;
1384
1385	/* Append the token to the queue. */
1386
1387	if (!ENQUEUE(parser, parser->tokens, token)) {
1388	yaml_token_delete(&token);
1389	return 0;
1390	}
1391
1392	return 1;
1393	}
1394
1395	/*
1396	* Produce the DOCUMENT-START or DOCUMENT-END token.
1397	*/
1398
1399	static int
1400	yaml_parser_fetch_document_indicator(yaml_parser_t *parser,
1401	yaml_token_type_t type)
1402	{
1403	yaml_mark_t start_mark, end_mark;
1404	yaml_token_t token;
1405
1406	/* Reset the indentation level. */
1407
1408	if (!yaml_parser_unroll_indent(parser, -1))
1409	return 0;
1410
1411	/* Reset simple keys. */
1412
1413	if (!yaml_parser_remove_simple_key(parser))
1414	return 0;
1415
1416	parser->simple_key_allowed = 0;
1417
1418	/* Consume the token. */
1419
1420	start_mark = parser->mark;
1421
1422	SKIP(parser);
1423	SKIP(parser);
1424	SKIP(parser);
1425
1426	end_mark = parser->mark;
1427
1428	/* Create the DOCUMENT-START or DOCUMENT-END token. */
1429
1430	TOKEN_INIT(token, type, start_mark, end_mark);
1431
1432	/* Append the token to the queue. */
1433
1434	if (!ENQUEUE(parser, parser->tokens, token))
1435	return 0;
1436
1437	return 1;
1438	}
1439
1440	/*
1441	* Produce the FLOW-SEQUENCE-START or FLOW-MAPPING-START token.
1442	*/
1443
1444	static int
1445	yaml_parser_fetch_flow_collection_start(yaml_parser_t *parser,
1446	yaml_token_type_t type)
1447	{
1448	yaml_mark_t start_mark, end_mark;
1449	yaml_token_t token;
1450
1451	/* The indicators '[' and '{' may start a simple key. */
1452
1453	if (!yaml_parser_save_simple_key(parser))
1454	return 0;
1455
1456	/* Increase the flow level. */
1457
1458	if (!yaml_parser_increase_flow_level(parser))
1459	return 0;
1460
1461	/* A simple key may follow the indicators '[' and '{'. */
1462
1463	parser->simple_key_allowed = 1;
1464
1465	/* Consume the token. */
1466
1467	start_mark = parser->mark;
1468	SKIP(parser);
1469	end_mark = parser->mark;
1470
1471	/* Create the FLOW-SEQUENCE-START of FLOW-MAPPING-START token. */
1472
1473	TOKEN_INIT(token, type, start_mark, end_mark);
1474
1475	/* Append the token to the queue. */
1476
1477	if (!ENQUEUE(parser, parser->tokens, token))
1478	return 0;
1479
1480	return 1;
1481	}
1482
1483	/*
1484	* Produce the FLOW-SEQUENCE-END or FLOW-MAPPING-END token.
1485	*/
1486
1487	static int
1488	yaml_parser_fetch_flow_collection_end(yaml_parser_t *parser,
1489	yaml_token_type_t type)
1490	{
1491	yaml_mark_t start_mark, end_mark;
1492	yaml_token_t token;
1493
1494	/* Reset any potential simple key on the current flow level. */
1495
1496	if (!yaml_parser_remove_simple_key(parser))
1497	return 0;
1498
1499	/* Decrease the flow level. */
1500
1501	if (!yaml_parser_decrease_flow_level(parser))
1502	return 0;
1503
1504	/* No simple keys after the indicators ']' and '}'. */
1505
1506	parser->simple_key_allowed = 0;
1507
1508	/* Consume the token. */
1509
1510	start_mark = parser->mark;
1511	SKIP(parser);
1512	end_mark = parser->mark;
1513
1514	/* Create the FLOW-SEQUENCE-END of FLOW-MAPPING-END token. */
1515
1516	TOKEN_INIT(token, type, start_mark, end_mark);
1517
1518	/* Append the token to the queue. */
1519
1520	if (!ENQUEUE(parser, parser->tokens, token))
1521	return 0;
1522
1523	return 1;
1524	}
1525
1526	/*
1527	* Produce the FLOW-ENTRY token.
1528	*/
1529
1530	static int
1531	yaml_parser_fetch_flow_entry(yaml_parser_t *parser)
1532	{
1533	yaml_mark_t start_mark, end_mark;
1534	yaml_token_t token;
1535
1536	/* Reset any potential simple keys on the current flow level. */
1537
1538	if (!yaml_parser_remove_simple_key(parser))
1539	return 0;
1540
1541	/* Simple keys are allowed after ','. */
1542
1543	parser->simple_key_allowed = 1;
1544
1545	/* Consume the token. */
1546
1547	start_mark = parser->mark;
1548	SKIP(parser);
1549	end_mark = parser->mark;
1550
1551	/* Create the FLOW-ENTRY token and append it to the queue. */
1552
1553	TOKEN_INIT(token, YAML_FLOW_ENTRY_TOKEN, start_mark, end_mark);
1554
1555	if (!ENQUEUE(parser, parser->tokens, token))
1556	return 0;
1557
1558	return 1;
1559	}
1560
1561	/*
1562	* Produce the BLOCK-ENTRY token.
1563	*/
1564
1565	static int
1566	yaml_parser_fetch_block_entry(yaml_parser_t *parser)
1567	{
1568	yaml_mark_t start_mark, end_mark;
1569	yaml_token_t token;
1570
1571	/* Check if the scanner is in the block context. */
1572
1573	if (!parser->flow_level)
1574	{
1575	/* Check if we are allowed to start a new entry. */
1576
1577	if (!parser->simple_key_allowed) {
1578	return yaml_parser_set_scanner_error(parser, NULL, parser->mark,
1579	"block sequence entries are not allowed in this context");
1580	}
1581
1582	/* Add the BLOCK-SEQUENCE-START token if needed. */
1583
1584	if (!yaml_parser_roll_indent(parser, parser->mark.column, -1,
1585	YAML_BLOCK_SEQUENCE_START_TOKEN, parser->mark))
1586	return 0;
1587	}
1588	else
1589	{
1590	/*
1591	* It is an error for the '-' indicator to occur in the flow context,
1592	* but we let the Parser detect and report about it because the Parser
1593	* is able to point to the context.
1594	*/
1595	}
1596
1597	/* Reset any potential simple keys on the current flow level. */
1598
1599	if (!yaml_parser_remove_simple_key(parser))
1600	return 0;
1601
1602	/* Simple keys are allowed after '-'. */
1603
1604	parser->simple_key_allowed = 1;
1605
1606	/* Consume the token. */
1607
1608	start_mark = parser->mark;
1609	SKIP(parser);
1610	end_mark = parser->mark;
1611
1612	/* Create the BLOCK-ENTRY token and append it to the queue. */
1613
1614	TOKEN_INIT(token, YAML_BLOCK_ENTRY_TOKEN, start_mark, end_mark);
1615
1616	if (!ENQUEUE(parser, parser->tokens, token))
1617	return 0;
1618
1619	return 1;
1620	}
1621
1622	/*
1623	* Produce the KEY token.
1624	*/
1625
1626	static int
1627	yaml_parser_fetch_key(yaml_parser_t *parser)
1628	{
1629	yaml_mark_t start_mark, end_mark;
1630	yaml_token_t token;
1631
1632	/* In the block context, additional checks are required. */
1633
1634	if (!parser->flow_level)
1635	{
1636	/* Check if we are allowed to start a new key (not nessesary simple). */
1637
1638	if (!parser->simple_key_allowed) {
1639	return yaml_parser_set_scanner_error(parser, NULL, parser->mark,
1640	"mapping keys are not allowed in this context");
1641	}
1642
1643	/* Add the BLOCK-MAPPING-START token if needed. */
1644
1645	if (!yaml_parser_roll_indent(parser, parser->mark.column, -1,
1646	YAML_BLOCK_MAPPING_START_TOKEN, parser->mark))
1647	return 0;
1648	}
1649
1650	/* Reset any potential simple keys on the current flow level. */
1651
1652	if (!yaml_parser_remove_simple_key(parser))
1653	return 0;
1654
1655	/* Simple keys are allowed after '?' in the block context. */
1656
1657	parser->simple_key_allowed = (!parser->flow_level);
1658
1659	/* Consume the token. */
1660
1661	start_mark = parser->mark;
1662	SKIP(parser);
1663	end_mark = parser->mark;
1664
1665	/* Create the KEY token and append it to the queue. */
1666
1667	TOKEN_INIT(token, YAML_KEY_TOKEN, start_mark, end_mark);
1668
1669	if (!ENQUEUE(parser, parser->tokens, token))
1670	return 0;
1671
1672	return 1;
1673	}
1674
1675	/*
1676	* Produce the VALUE token.
1677	*/
1678
1679	static int
1680	yaml_parser_fetch_value(yaml_parser_t *parser)
1681	{
1682	yaml_mark_t start_mark, end_mark;
1683	yaml_token_t token;
1684	yaml_simple_key_t *simple_key = parser->simple_keys.top-1;
1685
1686	/* Have we found a simple key? */
1687
1688	if (simple_key->possible)
1689	{
1690
1691	/* Create the KEY token and insert it into the queue. */
1692
1693	TOKEN_INIT(token, YAML_KEY_TOKEN, simple_key->mark, simple_key->mark);
1694
1695	if (!QUEUE_INSERT(parser, parser->tokens,
1696	simple_key->token_number - parser->tokens_parsed, token))
1697	return 0;
1698
1699	/* In the block context, we may need to add the BLOCK-MAPPING-START token. */
1700
1701	if (!yaml_parser_roll_indent(parser, simple_key->mark.column,
1702	simple_key->token_number,
1703	YAML_BLOCK_MAPPING_START_TOKEN, simple_key->mark))
1704	return 0;
1705
1706	/* Remove the simple key. */
1707
1708	simple_key->possible = 0;
1709
1710	/* A simple key cannot follow another simple key. */
1711
1712	parser->simple_key_allowed = 0;
1713	}
1714	else
1715	{
1716	/* The ':' indicator follows a complex key. */
1717
1718	/* In the block context, extra checks are required. */
1719
1720	if (!parser->flow_level)
1721	{
1722	/* Check if we are allowed to start a complex value. */
1723
1724	if (!parser->simple_key_allowed) {
1725	return yaml_parser_set_scanner_error(parser, NULL, parser->mark,
1726	"mapping values are not allowed in this context");
1727	}
1728
1729	/* Add the BLOCK-MAPPING-START token if needed. */
1730
1731	if (!yaml_parser_roll_indent(parser, parser->mark.column, -1,
1732	YAML_BLOCK_MAPPING_START_TOKEN, parser->mark))
1733	return 0;
1734	}
1735
1736	/* Simple keys after ':' are allowed in the block context. */
1737
1738	parser->simple_key_allowed = (!parser->flow_level);
1739	}
1740
1741	/* Consume the token. */
1742
1743	start_mark = parser->mark;
1744	SKIP(parser);
1745	end_mark = parser->mark;
1746
1747	/* Create the VALUE token and append it to the queue. */
1748
1749	TOKEN_INIT(token, YAML_VALUE_TOKEN, start_mark, end_mark);
1750
1751	if (!ENQUEUE(parser, parser->tokens, token))
1752	return 0;
1753
1754	return 1;
1755	}
1756
1757	/*
1758	* Produce the ALIAS or ANCHOR token.
1759	*/
1760
1761	static int
1762	yaml_parser_fetch_anchor(yaml_parser_t *parser, yaml_token_type_t type)
1763	{
1764	yaml_token_t token;
1765
1766	/* An anchor or an alias could be a simple key. */
1767
1768	if (!yaml_parser_save_simple_key(parser))
1769	return 0;
1770
1771	/* A simple key cannot follow an anchor or an alias. */
1772
1773	parser->simple_key_allowed = 0;
1774
1775	/* Create the ALIAS or ANCHOR token and append it to the queue. */
1776
1777	if (!yaml_parser_scan_anchor(parser, &token, type))
1778	return 0;
1779
1780	if (!ENQUEUE(parser, parser->tokens, token)) {
1781	yaml_token_delete(&token);
1782	return 0;
1783	}
1784	return 1;
1785	}
1786
1787	/*
1788	* Produce the TAG token.
1789	*/
1790
1791	static int
1792	yaml_parser_fetch_tag(yaml_parser_t *parser)
1793	{
1794	yaml_token_t token;
1795
1796	/* A tag could be a simple key. */
1797
1798	if (!yaml_parser_save_simple_key(parser))
1799	return 0;
1800
1801	/* A simple key cannot follow a tag. */
1802
1803	parser->simple_key_allowed = 0;
1804
1805	/* Create the TAG token and append it to the queue. */
1806
1807	if (!yaml_parser_scan_tag(parser, &token))
1808	return 0;
1809
1810	if (!ENQUEUE(parser, parser->tokens, token)) {
1811	yaml_token_delete(&token);
1812	return 0;
1813	}
1814
1815	return 1;
1816	}
1817
1818	/*
1819	* Produce the SCALAR(...,literal) or SCALAR(...,folded) tokens.
1820	*/
1821
1822	static int
1823	yaml_parser_fetch_block_scalar(yaml_parser_t *parser, int literal)
1824	{
1825	yaml_token_t token;
1826
1827	/* Remove any potential simple keys. */
1828
1829	if (!yaml_parser_remove_simple_key(parser))
1830	return 0;
1831
1832	/* A simple key may follow a block scalar. */
1833
1834	parser->simple_key_allowed = 1;
1835
1836	/* Create the SCALAR token and append it to the queue. */
1837
1838	if (!yaml_parser_scan_block_scalar(parser, &token, literal))
1839	return 0;
1840
1841	if (!ENQUEUE(parser, parser->tokens, token)) {
1842	yaml_token_delete(&token);
1843	return 0;
1844	}
1845
1846	return 1;
1847	}
1848
1849	/*
1850	* Produce the SCALAR(...,single-quoted) or SCALAR(...,double-quoted) tokens.
1851	*/
1852
1853	static int
1854	yaml_parser_fetch_flow_scalar(yaml_parser_t *parser, int single)
1855	{
1856	yaml_token_t token;
1857
1858	/* A plain scalar could be a simple key. */
1859
1860	if (!yaml_parser_save_simple_key(parser))
1861	return 0;
1862
1863	/* A simple key cannot follow a flow scalar. */
1864
1865	parser->simple_key_allowed = 0;
1866
1867	/* Create the SCALAR token and append it to the queue. */
1868
1869	if (!yaml_parser_scan_flow_scalar(parser, &token, single))
1870	return 0;
1871
1872	if (!ENQUEUE(parser, parser->tokens, token)) {
1873	yaml_token_delete(&token);
1874	return 0;
1875	}
1876
1877	return 1;
1878	}
1879
1880	/*
1881	* Produce the SCALAR(...,plain) token.
1882	*/
1883
1884	static int
1885	yaml_parser_fetch_plain_scalar(yaml_parser_t *parser)
1886	{
1887	yaml_token_t token;
1888
1889	/* A plain scalar could be a simple key. */
1890
1891	if (!yaml_parser_save_simple_key(parser))
1892	return 0;
1893
1894	/* A simple key cannot follow a flow scalar. */
1895
1896	parser->simple_key_allowed = 0;
1897
1898	/* Create the SCALAR token and append it to the queue. */
1899
1900	if (!yaml_parser_scan_plain_scalar(parser, &token))
1901	return 0;
1902
1903	if (!ENQUEUE(parser, parser->tokens, token)) {
1904	yaml_token_delete(&token);
1905	return 0;
1906	}
1907
1908	return 1;
1909	}
1910
1911	/*
1912	* Eat whitespaces and comments until the next token is found.
1913	*/
1914
1915	static int
1916	yaml_parser_scan_to_next_token(yaml_parser_t *parser)
1917	{
1918	/* Until the next token is not found. */
1919
1920	while (1)
1921	{
1922	/* Allow the BOM mark to start a line. */
1923
1924	if (!CACHE(parser, 1)) return 0;
1925
1926	if (parser->mark.column == 0 && IS_BOM(parser->buffer))
1927	SKIP(parser);
1928
1929	/*
1930	* Eat whitespaces.
1931	*
1932	* Tabs are allowed:
1933	*
1934	* - in the flow context;
1935	* - in the block context, but not at the beginning of the line or
1936	* after '-', '?', or ':' (complex value).
1937	*/
1938
1939	if (!CACHE(parser, 1)) return 0;
1940
1941	while (CHECK(parser->buffer,' ') \|\|
1942	((parser->flow_level \|\| !parser->simple_key_allowed) &&
1943	CHECK(parser->buffer, '\t'))) {
1944	SKIP(parser);
1945	if (!CACHE(parser, 1)) return 0;
1946	}
1947
1948	/* Eat a comment until a line break. */
1949
1950	if (CHECK(parser->buffer, '#')) {
1951	while (!IS_BREAKZ(parser->buffer)) {
1952	SKIP(parser);
1953	if (!CACHE(parser, 1)) return 0;
1954	}
1955	}
1956
1957	/* If it is a line break, eat it. */
1958
1959	if (IS_BREAK(parser->buffer))
1960	{
1961	if (!CACHE(parser, 2)) return 0;
1962	SKIP_LINE(parser);
1963
1964	/* In the block context, a new line may start a simple key. */
1965
1966	if (!parser->flow_level) {
1967	parser->simple_key_allowed = 1;
1968	}
1969	}
1970	else
1971	{
1972	/* We have found a token. */
1973
1974	break;
1975	}
1976	}
1977
1978	return 1;
1979	}
1980
1981	/*
1982	* Scan a YAML-DIRECTIVE or TAG-DIRECTIVE token.
1983	*
1984	* Scope:
1985	* %YAML 1.1 # a comment \n
1986	* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1987	* %TAG !yaml! tag:yaml.org,2002: \n
1988	* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1989	*/
1990
1991	int
1992	yaml_parser_scan_directive(yaml_parser_t parser, yaml_token_t token)
1993	{
1994	yaml_mark_t start_mark, end_mark;
1995	yaml_char_t *name = NULL;
1996	int major, minor;
1997	yaml_char_t handle = NULL, prefix = NULL;
1998
1999	/* Eat '%'. */
2000
2001	start_mark = parser->mark;
2002
2003	SKIP(parser);
2004
2005	/* Scan the directive name. */
2006
2007	if (!yaml_parser_scan_directive_name(parser, start_mark, &name))
2008	goto error;
2009
2010	/* Is it a YAML directive? */
2011
2012	if (strcmp((char *)name, "YAML") == 0)
2013	{
2014	/* Scan the VERSION directive value. */
2015
2016	if (!yaml_parser_scan_version_directive_value(parser, start_mark,
2017	&major, &minor))
2018	goto error;
2019
2020	end_mark = parser->mark;
2021
2022	/* Create a VERSION-DIRECTIVE token. */
2023
2024	VERSION_DIRECTIVE_TOKEN_INIT(*token, major, minor,
2025	start_mark, end_mark);
2026	}
2027
2028	/* Is it a TAG directive? */
2029
2030	else if (strcmp((char *)name, "TAG") == 0)
2031	{
2032	/* Scan the TAG directive value. */
2033
2034	if (!yaml_parser_scan_tag_directive_value(parser, start_mark,
2035	&handle, &prefix))
2036	goto error;
2037
2038	end_mark = parser->mark;
2039
2040	/* Create a TAG-DIRECTIVE token. */
2041
2042	TAG_DIRECTIVE_TOKEN_INIT(*token, handle, prefix,
2043	start_mark, end_mark);
2044	}
2045
2046	/* Unknown directive. */
2047
2048	else
2049	{
2050	yaml_parser_set_scanner_error(parser, "while scanning a directive",
2051	start_mark, "found uknown directive name");
2052	goto error;
2053	}
2054
2055	/* Eat the rest of the line including any comments. */
2056
2057	if (!CACHE(parser, 1)) goto error;
2058
2059	while (IS_BLANK(parser->buffer)) {
2060	SKIP(parser);
2061	if (!CACHE(parser, 1)) goto error;
2062	}
2063
2064	if (CHECK(parser->buffer, '#')) {
2065	while (!IS_BREAKZ(parser->buffer)) {
2066	SKIP(parser);
2067	if (!CACHE(parser, 1)) goto error;
2068	}
2069	}
2070
2071	/* Check if we are at the end of the line. */
2072
2073	if (!IS_BREAKZ(parser->buffer)) {
2074	yaml_parser_set_scanner_error(parser, "while scanning a directive",
2075	start_mark, "did not found expected comment or line break");
2076	goto error;
2077	}
2078
2079	/* Eat a line break. */
2080
2081	if (IS_BREAK(parser->buffer)) {
2082	if (!CACHE(parser, 2)) goto error;
2083	SKIP_LINE(parser);
2084	}
2085
2086	yaml_free(name);
2087
2088	return 1;
2089
2090	error:
2091	yaml_free(prefix);
2092	yaml_free(handle);
2093	yaml_free(name);
2094	return 0;
2095	}
2096
2097	/*
2098	* Scan the directive name.
2099	*
2100	* Scope:
2101	* %YAML 1.1 # a comment \n
2102	* ^^^^
2103	* %TAG !yaml! tag:yaml.org,2002: \n
2104	* ^^^
2105	*/
2106
2107	static int
2108	yaml_parser_scan_directive_name(yaml_parser_t *parser,
2109	yaml_mark_t start_mark, yaml_char_t **name)
2110	{
2111	yaml_string_t string = NULL_STRING;
2112
2113	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
2114
2115	/* Consume the directive name. */
2116
2117	if (!CACHE(parser, 1)) goto error;
2118
2119	while (IS_ALPHA(parser->buffer))
2120	{
2121	if (!READ(parser, string)) goto error;
2122	if (!CACHE(parser, 1)) goto error;
2123	}
2124
2125	/* Check if the name is empty. */
2126
2127	if (string.start == string.pointer) {
2128	yaml_parser_set_scanner_error(parser, "while scanning a directive",
2129	start_mark, "cannot found expected directive name");
2130	goto error;
2131	}
2132
2133	/* Check for an blank character after the name. */
2134
2135	if (!IS_BLANKZ(parser->buffer)) {
2136	yaml_parser_set_scanner_error(parser, "while scanning a directive",
2137	start_mark, "found unexpected non-alphabetical character");
2138	goto error;
2139	}
2140
2141	*name = string.start;
2142
2143	return 1;
2144
2145	error:
2146	STRING_DEL(parser, string);
2147	return 0;
2148	}
2149
2150	/*
2151	* Scan the value of VERSION-DIRECTIVE.
2152	*
2153	* Scope:
2154	* %YAML 1.1 # a comment \n
2155	* ^^^^^^
2156	*/
2157
2158	static int
2159	yaml_parser_scan_version_directive_value(yaml_parser_t *parser,
2160	yaml_mark_t start_mark, int major, int minor)
2161	{
2162	/* Eat whitespaces. */
2163
2164	if (!CACHE(parser, 1)) return 0;
2165
2166	while (IS_BLANK(parser->buffer)) {
2167	SKIP(parser);
2168	if (!CACHE(parser, 1)) return 0;
2169	}
2170
2171	/* Consume the major version number. */
2172
2173	if (!yaml_parser_scan_version_directive_number(parser, start_mark, major))
2174	return 0;
2175
2176	/* Eat '.'. */
2177
2178	if (!CHECK(parser->buffer, '.')) {
2179	return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive",
2180	start_mark, "did not find expected digit or '.' character");
2181	}
2182
2183	SKIP(parser);
2184
2185	/* Consume the minor version number. */
2186
2187	if (!yaml_parser_scan_version_directive_number(parser, start_mark, minor))
2188	return 0;
2189
2190	return 1;
2191	}
2192
2193	#define MAX_NUMBER_LENGTH 9
2194
2195	/*
2196	* Scan the version number of VERSION-DIRECTIVE.
2197	*
2198	* Scope:
2199	* %YAML 1.1 # a comment \n
2200	* ^
2201	* %YAML 1.1 # a comment \n
2202	* ^
2203	*/
2204
2205	static int
2206	yaml_parser_scan_version_directive_number(yaml_parser_t *parser,
2207	yaml_mark_t start_mark, int *number)
2208	{
2209	int value = 0;
2210	size_t length = 0;
2211
2212	/* Repeat while the next character is digit. */
2213
2214	if (!CACHE(parser, 1)) return 0;
2215
2216	while (IS_DIGIT(parser->buffer))
2217	{
2218	/* Check if the number is too long. */
2219
2220	if (++length > MAX_NUMBER_LENGTH) {
2221	return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive",
2222	start_mark, "found extremely long version number");
2223	}
2224
2225	value = value*10 + AS_DIGIT(parser->buffer);
2226
2227	SKIP(parser);
2228
2229	if (!CACHE(parser, 1)) return 0;
2230	}
2231
2232	/* Check if the number was present. */
2233
2234	if (!length) {
2235	return yaml_parser_set_scanner_error(parser, "while scanning a %YAML directive",
2236	start_mark, "did not find expected version number");
2237	}
2238
2239	*number = value;
2240
2241	return 1;
2242	}
2243
2244	/*
2245	* Scan the value of a TAG-DIRECTIVE token.
2246	*
2247	* Scope:
2248	* %TAG !yaml! tag:yaml.org,2002: \n
2249	* ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2250	*/
2251
2252	static int
2253	yaml_parser_scan_tag_directive_value(yaml_parser_t *parser,
2254	yaml_mark_t start_mark, yaml_char_t handle, yaml_char_t prefix)
2255	{
2256	yaml_char_t *handle_value = NULL;
2257	yaml_char_t *prefix_value = NULL;
2258
2259	/* Eat whitespaces. */
2260
2261	if (!CACHE(parser, 1)) goto error;
2262
2263	while (IS_BLANK(parser->buffer)) {
2264	SKIP(parser);
2265	if (!CACHE(parser, 1)) goto error;
2266	}
2267
2268	/* Scan a handle. */
2269
2270	if (!yaml_parser_scan_tag_handle(parser, 1, start_mark, &handle_value))
2271	goto error;
2272
2273	/* Expect a whitespace. */
2274
2275	if (!CACHE(parser, 1)) goto error;
2276
2277	if (!IS_BLANK(parser->buffer)) {
2278	yaml_parser_set_scanner_error(parser, "while scanning a %TAG directive",
2279	start_mark, "did not find expected whitespace");
2280	goto error;
2281	}
2282
2283	/* Eat whitespaces. */
2284
2285	while (IS_BLANK(parser->buffer)) {
2286	SKIP(parser);
2287	if (!CACHE(parser, 1)) goto error;
2288	}
2289
2290	/* Scan a prefix. */
2291
2292	if (!yaml_parser_scan_tag_uri(parser, 1, NULL, start_mark, &prefix_value))
2293	goto error;
2294
2295	/* Expect a whitespace or line break. */
2296
2297	if (!CACHE(parser, 1)) goto error;
2298
2299	if (!IS_BLANKZ(parser->buffer)) {
2300	yaml_parser_set_scanner_error(parser, "while scanning a %TAG directive",
2301	start_mark, "did not find expected whitespace or line break");
2302	goto error;
2303	}
2304
2305	*handle = handle_value;
2306	*prefix = prefix_value;
2307
2308	return 1;
2309
2310	error:
2311	yaml_free(handle_value);
2312	yaml_free(prefix_value);
2313	return 0;
2314	}
2315
2316	static int
2317	yaml_parser_scan_anchor(yaml_parser_t parser, yaml_token_t token,
2318	yaml_token_type_t type)
2319	{
2320	int length = 0;
2321	yaml_mark_t start_mark, end_mark;
2322	yaml_string_t string = NULL_STRING;
2323
2324	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
2325
2326	/* Eat the indicator character. */
2327
2328	start_mark = parser->mark;
2329
2330	SKIP(parser);
2331
2332	/* Consume the value. */
2333
2334	if (!CACHE(parser, 1)) goto error;
2335
2336	while (IS_ALPHA(parser->buffer)) {
2337	if (!READ(parser, string)) goto error;
2338	if (!CACHE(parser, 1)) goto error;
2339	length ++;
2340	}
2341
2342	end_mark = parser->mark;
2343
2344	/*
2345	* Check if length of the anchor is greater than 0 and it is followed by
2346	* a whitespace character or one of the indicators:
2347	*
2348	* '?', ':', ',', ']', '}', '%', '@', '`'.
2349	*/
2350
2351	if (!length \|\| !(IS_BLANKZ(parser->buffer) \|\| CHECK(parser->buffer, '?')
2352	\|\| CHECK(parser->buffer, ':') \|\| CHECK(parser->buffer, ',')
2353	\|\| CHECK(parser->buffer, ']') \|\| CHECK(parser->buffer, '}')
2354	\|\| CHECK(parser->buffer, '%') \|\| CHECK(parser->buffer, '@')
2355	\|\| CHECK(parser->buffer, '`'))) {
2356	yaml_parser_set_scanner_error(parser, type == YAML_ANCHOR_TOKEN ?
2357	"while scanning an anchor" : "while scanning an alias", start_mark,
2358	"did not find expected alphabetic or numeric character");
2359	goto error;
2360	}
2361
2362	/* Create a token. */
2363
2364	if (type == YAML_ANCHOR_TOKEN) {
2365	ANCHOR_TOKEN_INIT(*token, string.start, start_mark, end_mark);
2366	}
2367	else {
2368	ALIAS_TOKEN_INIT(*token, string.start, start_mark, end_mark);
2369	}
2370
2371	return 1;
2372
2373	error:
2374	STRING_DEL(parser, string);
2375	return 0;
2376	}
2377
2378	/*
2379	* Scan a TAG token.
2380	*/
2381
2382	static int
2383	yaml_parser_scan_tag(yaml_parser_t parser, yaml_token_t token)
2384	{
2385	yaml_char_t *handle = NULL;
2386	yaml_char_t *suffix = NULL;
2387	yaml_mark_t start_mark, end_mark;
2388
2389	start_mark = parser->mark;
2390
2391	/* Check if the tag is in the canonical form. */
2392
2393	if (!CACHE(parser, 2)) goto error;
2394
2395	if (CHECK_AT(parser->buffer, '<', 1))
2396	{
2397	/* Set the handle to '' */
2398
2399	handle = yaml_malloc(1);
2400	if (!handle) goto error;
2401	handle[0] = '\0';
2402
2403	/* Eat '!<' */
2404
2405	SKIP(parser);
2406	SKIP(parser);
2407
2408	/* Consume the tag value. */
2409
2410	if (!yaml_parser_scan_tag_uri(parser, 0, NULL, start_mark, &suffix))
2411	goto error;
2412
2413	/* Check for '>' and eat it. */
2414
2415	if (!CHECK(parser->buffer, '>')) {
2416	yaml_parser_set_scanner_error(parser, "while scanning a tag",
2417	start_mark, "did not find the expected '>'");
2418	goto error;
2419	}
2420
2421	SKIP(parser);
2422	}
2423	else
2424	{
2425	/* The tag has either the '!suffix' or the '!handle!suffix' form. */
2426
2427	/* First, try to scan a handle. */
2428
2429	if (!yaml_parser_scan_tag_handle(parser, 0, start_mark, &handle))
2430	goto error;
2431
2432	/* Check if it is, indeed, handle. */
2433
2434	if (handle[0] == '!' && handle[1] != '\0' && handle[strlen((char *)handle)-1] == '!')
2435	{
2436	/* Scan the suffix now. */
2437
2438	if (!yaml_parser_scan_tag_uri(parser, 0, NULL, start_mark, &suffix))
2439	goto error;
2440	}
2441	else
2442	{
2443	/* It wasn't a handle after all. Scan the rest of the tag. */
2444
2445	if (!yaml_parser_scan_tag_uri(parser, 0, handle, start_mark, &suffix))
2446	goto error;
2447
2448	/* Set the handle to '!'. */
2449
2450	yaml_free(handle);
2451	handle = yaml_malloc(2);
2452	if (!handle) goto error;
2453	handle[0] = '!';
2454	handle[1] = '\0';
2455
2456	/*
2457	* A special case: the '!' tag. Set the handle to '' and the
2458	* suffix to '!'.
2459	*/
2460
2461	if (suffix[0] == '\0') {
2462	yaml_char_t *tmp = handle;
2463	handle = suffix;
2464	suffix = tmp;
2465	}
2466	}
2467	}
2468
2469	/* Check the character which ends the tag. */
2470
2471	if (!CACHE(parser, 1)) goto error;
2472
2473	if (!IS_BLANKZ(parser->buffer)) {
2474	yaml_parser_set_scanner_error(parser, "while scanning a tag",
2475	start_mark, "did not found expected whitespace or line break");
2476	goto error;
2477	}
2478
2479	end_mark = parser->mark;
2480
2481	/* Create a token. */
2482
2483	TAG_TOKEN_INIT(*token, handle, suffix, start_mark, end_mark);
2484
2485	return 1;
2486
2487	error:
2488	yaml_free(handle);
2489	yaml_free(suffix);
2490	return 0;
2491	}
2492
2493	/*
2494	* Scan a tag handle.
2495	*/
2496
2497	static int
2498	yaml_parser_scan_tag_handle(yaml_parser_t *parser, int directive,
2499	yaml_mark_t start_mark, yaml_char_t **handle)
2500	{
2501	yaml_string_t string = NULL_STRING;
2502
2503	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
2504
2505	/* Check the initial '!' character. */
2506
2507	if (!CACHE(parser, 1)) goto error;
2508
2509	if (!CHECK(parser->buffer, '!')) {
2510	yaml_parser_set_scanner_error(parser, directive ?
2511	"while scanning a tag directive" : "while scanning a tag",
2512	start_mark, "did not find expected '!'");
2513	goto error;
2514	}
2515
2516	/* Copy the '!' character. */
2517
2518	if (!READ(parser, string)) goto error;
2519
2520	/* Copy all subsequent alphabetical and numerical characters. */
2521
2522	if (!CACHE(parser, 1)) goto error;
2523
2524	while (IS_ALPHA(parser->buffer))
2525	{
2526	if (!READ(parser, string)) goto error;
2527	if (!CACHE(parser, 1)) goto error;
2528	}
2529
2530	/* Check if the trailing character is '!' and copy it. */
2531
2532	if (CHECK(parser->buffer, '!'))
2533	{
2534	if (!READ(parser, string)) goto error;
2535	}
2536	else
2537	{
2538	/*
2539	* It's either the '!' tag or not really a tag handle. If it's a %TAG
2540	* directive, it's an error. If it's a tag token, it must be a part of
2541	* URI.
2542	*/
2543
2544	if (directive && !(string.start[0] == '!' && string.start[1] == '\0')) {
2545	yaml_parser_set_scanner_error(parser, "while parsing a tag directive",
2546	start_mark, "did not find expected '!'");
2547	goto error;
2548	}
2549	}
2550
2551	*handle = string.start;
2552
2553	return 1;
2554
2555	error:
2556	STRING_DEL(parser, string);
2557	return 0;
2558	}
2559
2560	/*
2561	* Scan a tag.
2562	*/
2563
2564	static int
2565	yaml_parser_scan_tag_uri(yaml_parser_t *parser, int directive,
2566	yaml_char_t head, yaml_mark_t start_mark, yaml_char_t *uri)
2567	{
2568	size_t length = head ? strlen((char *)head) : 0;
2569	yaml_string_t string = NULL_STRING;
2570
2571	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
2572
2573	/* Resize the string to include the head. */
2574
2575	while (string.end - string.start <= (int)length) {
2576	if (!yaml_string_extend(&string.start, &string.pointer, &string.end)) {
2577	parser->error = YAML_MEMORY_ERROR;
2578	goto error;
2579	}
2580	}
2581
2582	/*
2583	* Copy the head if needed.
2584	*
2585	* Note that we don't copy the leading '!' character.
2586	*/
2587
2588	if (length > 1) {
2589	memcpy(string.start, head+1, length-1);
2590	string.pointer += length-1;
2591	}
2592
2593	/* Scan the tag. */
2594
2595	if (!CACHE(parser, 1)) goto error;
2596
2597	/*
2598	* The set of characters that may appear in URI is as follows:
2599	*
2600	* '0'-'9', 'A'-'Z', 'a'-'z', '_', '-', ';', '/', '?', ':', '@', '&',
2601	* '=', '+', '$', ',', '.', '!', '~', '*', '\'', '(', ')', '[', ']',
2602	* '%'.
2603	*/
2604
2605	while (IS_ALPHA(parser->buffer) \|\| CHECK(parser->buffer, ';')
2606	\|\| CHECK(parser->buffer, '/') \|\| CHECK(parser->buffer, '?')
2607	\|\| CHECK(parser->buffer, ':') \|\| CHECK(parser->buffer, '@')
2608	\|\| CHECK(parser->buffer, '&') \|\| CHECK(parser->buffer, '=')
2609	\|\| CHECK(parser->buffer, '+') \|\| CHECK(parser->buffer, '$')
2610	\|\| CHECK(parser->buffer, ',') \|\| CHECK(parser->buffer, '.')
2611	\|\| CHECK(parser->buffer, '!') \|\| CHECK(parser->buffer, '~')
2612	\|\| CHECK(parser->buffer, '*') \|\| CHECK(parser->buffer, '\'')
2613	\|\| CHECK(parser->buffer, '(') \|\| CHECK(parser->buffer, ')')
2614	\|\| CHECK(parser->buffer, '[') \|\| CHECK(parser->buffer, ']')
2615	\|\| CHECK(parser->buffer, '%'))
2616	{
2617	/* Check if it is a URI-escape sequence. */
2618
2619	if (CHECK(parser->buffer, '%')) {
2620	if (!yaml_parser_scan_uri_escapes(parser,
2621	directive, start_mark, &string)) goto error;
2622	}
2623	else {
2624	if (!READ(parser, string)) goto error;
2625	}
2626
2627	length ++;
2628	if (!CACHE(parser, 1)) goto error;
2629	}
2630
2631	/* Check if the tag is non-empty. */
2632
2633	if (!length) {
2634	if (!STRING_EXTEND(parser, string))
2635	goto error;
2636
2637	yaml_parser_set_scanner_error(parser, directive ?
2638	"while parsing a %TAG directive" : "while parsing a tag",
2639	start_mark, "did not find expected tag URI");
2640	goto error;
2641	}
2642
2643	*uri = string.start;
2644
2645	return 1;
2646
2647	error:
2648	STRING_DEL(parser, string);
2649	return 0;
2650	}
2651
2652	/*
2653	* Decode an URI-escape sequence corresponding to a single UTF-8 character.
2654	*/
2655
2656	static int
2657	yaml_parser_scan_uri_escapes(yaml_parser_t *parser, int directive,
2658	yaml_mark_t start_mark, yaml_string_t *string)
2659	{
2660	int width = 0;
2661
2662	/* Decode the required number of characters. */
2663
2664	do {
2665
2666	unsigned char octet = 0;
2667
2668	/* Check for a URI-escaped octet. */
2669
2670	if (!CACHE(parser, 3)) return 0;
2671
2672	if (!(CHECK(parser->buffer, '%')
2673	&& IS_HEX_AT(parser->buffer, 1)
2674	&& IS_HEX_AT(parser->buffer, 2))) {
2675	return yaml_parser_set_scanner_error(parser, directive ?
2676	"while parsing a %TAG directive" : "while parsing a tag",
2677	start_mark, "did not find URI escaped octet");
2678	}
2679
2680	/* Get the octet. */
2681
2682	octet = (AS_HEX_AT(parser->buffer, 1) << 4) + AS_HEX_AT(parser->buffer, 2);
2683
2684	/* If it is the leading octet, determine the length of the UTF-8 sequence. */
2685
2686	if (!width)
2687	{
2688	width = (octet & 0x80) == 0x00 ? 1 :
2689	(octet & 0xE0) == 0xC0 ? 2 :
2690	(octet & 0xF0) == 0xE0 ? 3 :
2691	(octet & 0xF8) == 0xF0 ? 4 : 0;
2692	if (!width) {
2693	return yaml_parser_set_scanner_error(parser, directive ?
2694	"while parsing a %TAG directive" : "while parsing a tag",
2695	start_mark, "found an incorrect leading UTF-8 octet");
2696	}
2697	}
2698	else
2699	{
2700	/* Check if the trailing octet is correct. */
2701
2702	if ((octet & 0xC0) != 0x80) {
2703	return yaml_parser_set_scanner_error(parser, directive ?
2704	"while parsing a %TAG directive" : "while parsing a tag",
2705	start_mark, "found an incorrect trailing UTF-8 octet");
2706	}
2707	}
2708
2709	/* Copy the octet and move the pointers. */
2710
2711	*(string->pointer++) = octet;
2712	SKIP(parser);
2713	SKIP(parser);
2714	SKIP(parser);
2715
2716	} while (--width);
2717
2718	return 1;
2719	}
2720
2721	/*
2722	* Scan a block scalar.
2723	*/
2724
2725	static int
2726	yaml_parser_scan_block_scalar(yaml_parser_t parser, yaml_token_t token,
2727	int literal)
2728	{
2729	yaml_mark_t start_mark;
2730	yaml_mark_t end_mark;
2731	yaml_string_t string = NULL_STRING;
2732	yaml_string_t leading_break = NULL_STRING;
2733	yaml_string_t trailing_breaks = NULL_STRING;
2734	int chomping = 0;
2735	int increment = 0;
2736	int indent = 0;
2737	int leading_blank = 0;
2738	int trailing_blank = 0;
2739
2740	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
2741	if (!STRING_INIT(parser, leading_break, INITIAL_STRING_SIZE)) goto error;
2742	if (!STRING_INIT(parser, trailing_breaks, INITIAL_STRING_SIZE)) goto error;
2743
2744	/* Eat the indicator '\|' or '>'. */
2745
2746	start_mark = parser->mark;
2747
2748	SKIP(parser);
2749
2750	/* Scan the additional block scalar indicators. */
2751
2752	if (!CACHE(parser, 1)) goto error;
2753
2754	/* Check for a chomping indicator. */
2755
2756	if (CHECK(parser->buffer, '+') \|\| CHECK(parser->buffer, '-'))
2757	{
2758	/* Set the chomping method and eat the indicator. */
2759
2760	chomping = CHECK(parser->buffer, '+') ? +1 : -1;
2761
2762	SKIP(parser);
2763
2764	/* Check for an indentation indicator. */
2765
2766	if (!CACHE(parser, 1)) goto error;
2767
2768	if (IS_DIGIT(parser->buffer))
2769	{
2770	/* Check that the intendation is greater than 0. */
2771
2772	if (CHECK(parser->buffer, '0')) {
2773	yaml_parser_set_scanner_error(parser, "while scanning a block scalar",
2774	start_mark, "found an intendation indicator equal to 0");
2775	goto error;
2776	}
2777
2778	/* Get the intendation level and eat the indicator. */
2779
2780	increment = AS_DIGIT(parser->buffer);
2781
2782	SKIP(parser);
2783	}
2784	}
2785
2786	/* Do the same as above, but in the opposite order. */
2787
2788	else if (IS_DIGIT(parser->buffer))
2789	{
2790	if (CHECK(parser->buffer, '0')) {
2791	yaml_parser_set_scanner_error(parser, "while scanning a block scalar",
2792	start_mark, "found an intendation indicator equal to 0");
2793	goto error;
2794	}
2795
2796	increment = AS_DIGIT(parser->buffer);
2797
2798	SKIP(parser);
2799
2800	if (!CACHE(parser, 1)) goto error;
2801
2802	if (CHECK(parser->buffer, '+') \|\| CHECK(parser->buffer, '-')) {
2803	chomping = CHECK(parser->buffer, '+') ? +1 : -1;
2804
2805	SKIP(parser);
2806	}
2807	}
2808
2809	/* Eat whitespaces and comments to the end of the line. */
2810
2811	if (!CACHE(parser, 1)) goto error;
2812
2813	while (IS_BLANK(parser->buffer)) {
2814	SKIP(parser);
2815	if (!CACHE(parser, 1)) goto error;
2816	}
2817
2818	if (CHECK(parser->buffer, '#')) {
2819	while (!IS_BREAKZ(parser->buffer)) {
2820	SKIP(parser);
2821	if (!CACHE(parser, 1)) goto error;
2822	}
2823	}
2824
2825	/* Check if we are at the end of the line. */
2826
2827	if (!IS_BREAKZ(parser->buffer)) {
2828	yaml_parser_set_scanner_error(parser, "while scanning a block scalar",
2829	start_mark, "did not found expected comment or line break");
2830	goto error;
2831	}
2832
2833	/* Eat a line break. */
2834
2835	if (IS_BREAK(parser->buffer)) {
2836	if (!CACHE(parser, 2)) goto error;
2837	SKIP_LINE(parser);
2838	}
2839
2840	end_mark = parser->mark;
2841
2842	/* Set the intendation level if it was specified. */
2843
2844	if (increment) {
2845	indent = parser->indent >= 0 ? parser->indent+increment : increment;
2846	}
2847
2848	/* Scan the leading line breaks and determine the indentation level if needed. */
2849
2850	if (!yaml_parser_scan_block_scalar_breaks(parser, &indent, &trailing_breaks,
2851	start_mark, &end_mark)) goto error;
2852
2853	/* Scan the block scalar content. */
2854
2855	if (!CACHE(parser, 1)) goto error;
2856
2857	while ((int)parser->mark.column == indent && !IS_Z(parser->buffer))
2858	{
2859	/*
2860	* We are at the beginning of a non-empty line.
2861	*/
2862
2863	/* Is it a trailing whitespace? */
2864
2865	trailing_blank = IS_BLANK(parser->buffer);
2866
2867	/* Check if we need to fold the leading line break. */
2868
2869	if (!literal && (*leading_break.start == '\n')
2870	&& !leading_blank && !trailing_blank)
2871	{
2872	/* Do we need to join the lines by space? */
2873
2874	if (*trailing_breaks.start == '\0') {
2875	if (!STRING_EXTEND(parser, string)) goto error;
2876	*(string.pointer ++) = ' ';
2877	}
2878
2879	CLEAR(parser, leading_break);
2880	}
2881	else {
2882	if (!JOIN(parser, string, leading_break)) goto error;
2883	CLEAR(parser, leading_break);
2884	}
2885
2886	/* Append the remaining line breaks. */
2887
2888	if (!JOIN(parser, string, trailing_breaks)) goto error;
2889	CLEAR(parser, trailing_breaks);
2890
2891	/* Is it a leading whitespace? */
2892
2893	leading_blank = IS_BLANK(parser->buffer);
2894
2895	/* Consume the current line. */
2896
2897	while (!IS_BREAKZ(parser->buffer)) {
2898	if (!READ(parser, string)) goto error;
2899	if (!CACHE(parser, 1)) goto error;
2900	}
2901
2902	/* Consume the line break. */
2903
2904	if (!CACHE(parser, 2)) goto error;
2905
2906	if (!READ_LINE(parser, leading_break)) goto error;
2907
2908	/* Eat the following intendation spaces and line breaks. */
2909
2910	if (!yaml_parser_scan_block_scalar_breaks(parser,
2911	&indent, &trailing_breaks, start_mark, &end_mark)) goto error;
2912	}
2913
2914	/* Chomp the tail. */
2915
2916	if (chomping != -1) {
2917	if (!JOIN(parser, string, leading_break)) goto error;
2918	}
2919	if (chomping == 1) {
2920	if (!JOIN(parser, string, trailing_breaks)) goto error;
2921	}
2922
2923	/* Create a token. */
2924
2925	SCALAR_TOKEN_INIT(*token, string.start, string.pointer-string.start,
2926	literal ? YAML_LITERAL_SCALAR_STYLE : YAML_FOLDED_SCALAR_STYLE,
2927	start_mark, end_mark);
2928
2929	STRING_DEL(parser, leading_break);
2930	STRING_DEL(parser, trailing_breaks);
2931
2932	return 1;
2933
2934	error:
2935	STRING_DEL(parser, string);
2936	STRING_DEL(parser, leading_break);
2937	STRING_DEL(parser, trailing_breaks);
2938
2939	return 0;
2940	}
2941
2942	/*
2943	* Scan intendation spaces and line breaks for a block scalar. Determine the
2944	* intendation level if needed.
2945	*/
2946
2947	static int
2948	yaml_parser_scan_block_scalar_breaks(yaml_parser_t *parser,
2949	int indent, yaml_string_t breaks,
2950	yaml_mark_t start_mark, yaml_mark_t *end_mark)
2951	{
2952	int max_indent = 0;
2953
2954	*end_mark = parser->mark;
2955
2956	/* Eat the intendation spaces and line breaks. */
2957
2958	while (1)
2959	{
2960	/* Eat the intendation spaces. */
2961
2962	if (!CACHE(parser, 1)) return 0;
2963
2964	while ((!indent \|\| (int)parser->mark.column < indent)
2965	&& IS_SPACE(parser->buffer)) {
2966	SKIP(parser);
2967	if (!CACHE(parser, 1)) return 0;
2968	}
2969
2970	if ((int)parser->mark.column > max_indent)
2971	max_indent = (int)parser->mark.column;
2972
2973	/* Check for a tab character messing the intendation. */
2974
2975	if ((!indent \|\| (int)parser->mark.column < indent)
2976	&& IS_TAB(parser->buffer)) {
2977	return yaml_parser_set_scanner_error(parser, "while scanning a block scalar",
2978	start_mark, "found a tab character where an intendation space is expected");
2979	}
2980
2981	/* Have we found a non-empty line? */
2982
2983	if (!IS_BREAK(parser->buffer)) break;
2984
2985	/* Consume the line break. */
2986
2987	if (!CACHE(parser, 2)) return 0;
2988	if (!READ_LINE(parser, *breaks)) return 0;
2989	*end_mark = parser->mark;
2990	}
2991
2992	/* Determine the indentation level if needed. */
2993
2994	if (!*indent) {
2995	*indent = max_indent;
2996	if (*indent < parser->indent + 1)
2997	*indent = parser->indent + 1;
2998	if (*indent < 1)
2999	*indent = 1;
3000	}
3001
3002	return 1;
3003	}
3004
3005	/*
3006	* Scan a quoted scalar.
3007	*/
3008
3009	static int
3010	yaml_parser_scan_flow_scalar(yaml_parser_t parser, yaml_token_t token,
3011	int single)
3012	{
3013	yaml_mark_t start_mark;
3014	yaml_mark_t end_mark;
3015	yaml_string_t string = NULL_STRING;
3016	yaml_string_t leading_break = NULL_STRING;
3017	yaml_string_t trailing_breaks = NULL_STRING;
3018	yaml_string_t whitespaces = NULL_STRING;
3019	int leading_blanks;
3020
3021	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
3022	if (!STRING_INIT(parser, leading_break, INITIAL_STRING_SIZE)) goto error;
3023	if (!STRING_INIT(parser, trailing_breaks, INITIAL_STRING_SIZE)) goto error;
3024	if (!STRING_INIT(parser, whitespaces, INITIAL_STRING_SIZE)) goto error;
3025
3026	/* Eat the left quote. */
3027
3028	start_mark = parser->mark;
3029
3030	SKIP(parser);
3031
3032	/* Consume the content of the quoted scalar. */
3033
3034	while (1)
3035	{
3036	/* Check that there are no document indicators at the beginning of the line. */
3037
3038	if (!CACHE(parser, 4)) goto error;
3039
3040	if (parser->mark.column == 0 &&
3041	((CHECK_AT(parser->buffer, '-', 0) &&
3042	CHECK_AT(parser->buffer, '-', 1) &&
3043	CHECK_AT(parser->buffer, '-', 2)) \|\|
3044	(CHECK_AT(parser->buffer, '.', 0) &&
3045	CHECK_AT(parser->buffer, '.', 1) &&
3046	CHECK_AT(parser->buffer, '.', 2))) &&
3047	IS_BLANKZ_AT(parser->buffer, 3))
3048	{
3049	yaml_parser_set_scanner_error(parser, "while scanning a quoted scalar",
3050	start_mark, "found unexpected document indicator");
3051	goto error;
3052	}
3053
3054	/* Check for EOF. */
3055
3056	if (IS_Z(parser->buffer)) {
3057	yaml_parser_set_scanner_error(parser, "while scanning a quoted scalar",
3058	start_mark, "found unexpected end of stream");
3059	goto error;
3060	}
3061
3062	/* Consume non-blank characters. */
3063
3064	if (!CACHE(parser, 2)) goto error;
3065
3066	leading_blanks = 0;
3067
3068	while (!IS_BLANKZ(parser->buffer))
3069	{
3070	/* Check for an escaped single quote. */
3071
3072	if (single && CHECK_AT(parser->buffer, '\'', 0)
3073	&& CHECK_AT(parser->buffer, '\'', 1))
3074	{
3075	if (!STRING_EXTEND(parser, string)) goto error;
3076	*(string.pointer++) = '\'';
3077	SKIP(parser);
3078	SKIP(parser);
3079	}
3080
3081	/* Check for the right quote. */
3082
3083	else if (CHECK(parser->buffer, single ? '\'' : '"'))
3084	{
3085	break;
3086	}
3087
3088	/* Check for an escaped line break. */
3089
3090	else if (!single && CHECK(parser->buffer, '\\')
3091	&& IS_BREAK_AT(parser->buffer, 1))
3092	{
3093	if (!CACHE(parser, 3)) goto error;
3094	SKIP(parser);
3095	SKIP_LINE(parser);
3096	leading_blanks = 1;
3097	break;
3098	}
3099
3100	/* Check for an escape sequence. */
3101
3102	else if (!single && CHECK(parser->buffer, '\\'))
3103	{
3104	size_t code_length = 0;
3105
3106	if (!STRING_EXTEND(parser, string)) goto error;
3107
3108	/* Check the escape character. */
3109
3110	switch (parser->buffer.pointer[1])
3111	{
3112	case '0':
3113	*(string.pointer++) = '\0';
3114	break;
3115
3116	case 'a':
3117	*(string.pointer++) = '\x07';
3118	break;
3119
3120	case 'b':
3121	*(string.pointer++) = '\x08';
3122	break;
3123
3124	case 't':
3125	case '\t':
3126	*(string.pointer++) = '\x09';
3127	break;
3128
3129	case 'n':
3130	*(string.pointer++) = '\x0A';
3131	break;
3132
3133	case 'v':
3134	*(string.pointer++) = '\x0B';
3135	break;
3136
3137	case 'f':
3138	*(string.pointer++) = '\x0C';
3139	break;
3140
3141	case 'r':
3142	*(string.pointer++) = '\x0D';
3143	break;
3144
3145	case 'e':
3146	*(string.pointer++) = '\x1B';
3147	break;
3148
3149	case ' ':
3150	*(string.pointer++) = '\x20';
3151	break;
3152
3153	case '"':
3154	*(string.pointer++) = '"';
3155	break;
3156
3157	case '\'':
3158	*(string.pointer++) = '\'';
3159	break;
3160
3161	case '\\':
3162	*(string.pointer++) = '\\';
3163	break;
3164
3165	case 'N': /* NEL (#x85) */
3166	*(string.pointer++) = '\xC2';
3167	*(string.pointer++) = '\x85';
3168	break;
3169
3170	case '_': /* #xA0 */
3171	*(string.pointer++) = '\xC2';
3172	*(string.pointer++) = '\xA0';
3173	break;
3174
3175	case 'L': /* LS (#x2028) */
3176	*(string.pointer++) = '\xE2';
3177	*(string.pointer++) = '\x80';
3178	*(string.pointer++) = '\xA8';
3179	break;
3180
3181	case 'P': /* PS (#x2029) */
3182	*(string.pointer++) = '\xE2';
3183	*(string.pointer++) = '\x80';
3184	*(string.pointer++) = '\xA9';
3185	break;
3186
3187	case 'x':
3188	code_length = 2;
3189	break;
3190
3191	case 'u':
3192	code_length = 4;
3193	break;
3194
3195	case 'U':
3196	code_length = 8;
3197	break;
3198
3199	default:
3200	yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar",
3201	start_mark, "found unknown escape character");
3202	goto error;
3203	}
3204
3205	SKIP(parser);
3206	SKIP(parser);
3207
3208	/* Consume an arbitrary escape code. */
3209
3210	if (code_length)
3211	{
3212	unsigned int value = 0;
3213	size_t k;
3214
3215	/* Scan the character value. */
3216
3217	if (!CACHE(parser, code_length)) goto error;
3218
3219	for (k = 0; k < code_length; k ++) {
3220	if (!IS_HEX_AT(parser->buffer, k)) {
3221	yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar",
3222	start_mark, "did not find expected hexdecimal number");
3223	goto error;
3224	}
3225	value = (value << 4) + AS_HEX_AT(parser->buffer, k);
3226	}
3227
3228	/* Check the value and write the character. */
3229
3230	if ((value >= 0xD800 && value <= 0xDFFF) \|\| value > 0x10FFFF) {
3231	yaml_parser_set_scanner_error(parser, "while parsing a quoted scalar",
3232	start_mark, "found invalid Unicode character escape code");
3233	goto error;
3234	}
3235
3236	if (value <= 0x7F) {
3237	*(string.pointer++) = value;
3238	}
3239	else if (value <= 0x7FF) {
3240	*(string.pointer++) = 0xC0 + (value >> 6);
3241	*(string.pointer++) = 0x80 + (value & 0x3F);
3242	}
3243	else if (value <= 0xFFFF) {
3244	*(string.pointer++) = 0xE0 + (value >> 12);
3245	*(string.pointer++) = 0x80 + ((value >> 6) & 0x3F);
3246	*(string.pointer++) = 0x80 + (value & 0x3F);
3247	}
3248	else {
3249	*(string.pointer++) = 0xF0 + (value >> 18);
3250	*(string.pointer++) = 0x80 + ((value >> 12) & 0x3F);
3251	*(string.pointer++) = 0x80 + ((value >> 6) & 0x3F);
3252	*(string.pointer++) = 0x80 + (value & 0x3F);
3253	}
3254
3255	/* Advance the pointer. */
3256
3257	for (k = 0; k < code_length; k ++) {
3258	SKIP(parser);
3259	}
3260	}
3261	}
3262
3263	else
3264	{
3265	/* It is a non-escaped non-blank character. */
3266
3267	if (!READ(parser, string)) goto error;
3268	}
3269
3270	if (!CACHE(parser, 2)) goto error;
3271	}
3272
3273	/* Check if we are at the end of the scalar. */
3274
3275	if (CHECK(parser->buffer, single ? '\'' : '"'))
3276	break;
3277
3278	/* Consume blank characters. */
3279
3280	if (!CACHE(parser, 1)) goto error;
3281
3282	while (IS_BLANK(parser->buffer) \|\| IS_BREAK(parser->buffer))
3283	{
3284	if (IS_BLANK(parser->buffer))
3285	{
3286	/* Consume a space or a tab character. */
3287
3288	if (!leading_blanks) {
3289	if (!READ(parser, whitespaces)) goto error;
3290	}
3291	else {
3292	SKIP(parser);
3293	}
3294	}
3295	else
3296	{
3297	if (!CACHE(parser, 2)) goto error;
3298
3299	/* Check if it is a first line break. */
3300
3301	if (!leading_blanks)
3302	{
3303	CLEAR(parser, whitespaces);
3304	if (!READ_LINE(parser, leading_break)) goto error;
3305	leading_blanks = 1;
3306	}
3307	else
3308	{
3309	if (!READ_LINE(parser, trailing_breaks)) goto error;
3310	}
3311	}
3312	if (!CACHE(parser, 1)) goto error;
3313	}
3314
3315	/* Join the whitespaces or fold line breaks. */
3316
3317	if (leading_blanks)
3318	{
3319	/* Do we need to fold line breaks? */
3320
3321	if (leading_break.start[0] == '\n') {
3322	if (trailing_breaks.start[0] == '\0') {
3323	if (!STRING_EXTEND(parser, string)) goto error;
3324	*(string.pointer++) = ' ';
3325	}
3326	else {
3327	if (!JOIN(parser, string, trailing_breaks)) goto error;
3328	CLEAR(parser, trailing_breaks);
3329	}
3330	CLEAR(parser, leading_break);
3331	}
3332	else {
3333	if (!JOIN(parser, string, leading_break)) goto error;
3334	if (!JOIN(parser, string, trailing_breaks)) goto error;
3335	CLEAR(parser, leading_break);
3336	CLEAR(parser, trailing_breaks);
3337	}
3338	}
3339	else
3340	{
3341	if (!JOIN(parser, string, whitespaces)) goto error;
3342	CLEAR(parser, whitespaces);
3343	}
3344	}
3345
3346	/* Eat the right quote. */
3347
3348	SKIP(parser);
3349
3350	end_mark = parser->mark;
3351
3352	/* Create a token. */
3353
3354	SCALAR_TOKEN_INIT(*token, string.start, string.pointer-string.start,
3355	single ? YAML_SINGLE_QUOTED_SCALAR_STYLE : YAML_DOUBLE_QUOTED_SCALAR_STYLE,
3356	start_mark, end_mark);
3357
3358	STRING_DEL(parser, leading_break);
3359	STRING_DEL(parser, trailing_breaks);
3360	STRING_DEL(parser, whitespaces);
3361
3362	return 1;
3363
3364	error:
3365	STRING_DEL(parser, string);
3366	STRING_DEL(parser, leading_break);
3367	STRING_DEL(parser, trailing_breaks);
3368	STRING_DEL(parser, whitespaces);
3369
3370	return 0;
3371	}
3372
3373	/*
3374	* Scan a plain scalar.
3375	*/
3376
3377	static int
3378	yaml_parser_scan_plain_scalar(yaml_parser_t parser, yaml_token_t token)
3379	{
3380	yaml_mark_t start_mark;
3381	yaml_mark_t end_mark;
3382	yaml_string_t string = NULL_STRING;
3383	yaml_string_t leading_break = NULL_STRING;
3384	yaml_string_t trailing_breaks = NULL_STRING;
3385	yaml_string_t whitespaces = NULL_STRING;
3386	int leading_blanks = 0;
3387	int indent = parser->indent+1;
3388
3389	if (!STRING_INIT(parser, string, INITIAL_STRING_SIZE)) goto error;
3390	if (!STRING_INIT(parser, leading_break, INITIAL_STRING_SIZE)) goto error;
3391	if (!STRING_INIT(parser, trailing_breaks, INITIAL_STRING_SIZE)) goto error;
3392	if (!STRING_INIT(parser, whitespaces, INITIAL_STRING_SIZE)) goto error;
3393
3394	start_mark = end_mark = parser->mark;
3395
3396	/* Consume the content of the plain scalar. */
3397
3398	while (1)
3399	{
3400	/* Check for a document indicator. */
3401
3402	if (!CACHE(parser, 4)) goto error;
3403
3404	if (parser->mark.column == 0 &&
3405	((CHECK_AT(parser->buffer, '-', 0) &&
3406	CHECK_AT(parser->buffer, '-', 1) &&
3407	CHECK_AT(parser->buffer, '-', 2)) \|\|
3408	(CHECK_AT(parser->buffer, '.', 0) &&
3409	CHECK_AT(parser->buffer, '.', 1) &&
3410	CHECK_AT(parser->buffer, '.', 2))) &&
3411	IS_BLANKZ_AT(parser->buffer, 3)) break;
3412
3413	/* Check for a comment. */
3414
3415	if (CHECK(parser->buffer, '#'))
3416	break;
3417
3418	/* Consume non-blank characters. */
3419
3420	while (!IS_BLANKZ(parser->buffer))
3421	{
3422	/* Check for 'x:x' in the flow context. TODO: Fix the test "spec-08-13". */
3423
3424	if (parser->flow_level
3425	&& CHECK(parser->buffer, ':')
3426	&& !IS_BLANKZ_AT(parser->buffer, 1)) {
3427	yaml_parser_set_scanner_error(parser, "while scanning a plain scalar",
3428	start_mark, "found unexpected ':'");
3429	goto error;
3430	}
3431
3432	/* Check for indicators that may end a plain scalar. */
3433
3434	if ((CHECK(parser->buffer, ':') && IS_BLANKZ_AT(parser->buffer, 1))
3435	\|\| (parser->flow_level &&
3436	(CHECK(parser->buffer, ',') \|\| CHECK(parser->buffer, ':')
3437	\|\| CHECK(parser->buffer, '?') \|\| CHECK(parser->buffer, '[')
3438	\|\| CHECK(parser->buffer, ']') \|\| CHECK(parser->buffer, '{')
3439	\|\| CHECK(parser->buffer, '}'))))
3440	break;
3441
3442	/* Check if we need to join whitespaces and breaks. */
3443
3444	if (leading_blanks \|\| whitespaces.start != whitespaces.pointer)
3445	{
3446	if (leading_blanks)
3447	{
3448	/* Do we need to fold line breaks? */
3449
3450	if (leading_break.start[0] == '\n') {
3451	if (trailing_breaks.start[0] == '\0') {
3452	if (!STRING_EXTEND(parser, string)) goto error;
3453	*(string.pointer++) = ' ';
3454	}
3455	else {
3456	if (!JOIN(parser, string, trailing_breaks)) goto error;
3457	CLEAR(parser, trailing_breaks);
3458	}
3459	CLEAR(parser, leading_break);
3460	}
3461	else {
3462	if (!JOIN(parser, string, leading_break)) goto error;
3463	if (!JOIN(parser, string, trailing_breaks)) goto error;
3464	CLEAR(parser, leading_break);
3465	CLEAR(parser, trailing_breaks);
3466	}
3467
3468	leading_blanks = 0;
3469	}
3470	else
3471	{
3472	if (!JOIN(parser, string, whitespaces)) goto error;
3473	CLEAR(parser, whitespaces);
3474	}
3475	}
3476
3477	/* Copy the character. */
3478
3479	if (!READ(parser, string)) goto error;
3480
3481	end_mark = parser->mark;
3482
3483	if (!CACHE(parser, 2)) goto error;
3484	}
3485
3486	/* Is it the end? */
3487
3488	if (!(IS_BLANK(parser->buffer) \|\| IS_BREAK(parser->buffer)))
3489	break;
3490
3491	/* Consume blank characters. */
3492
3493	if (!CACHE(parser, 1)) goto error;
3494
3495	while (IS_BLANK(parser->buffer) \|\| IS_BREAK(parser->buffer))
3496	{
3497	if (IS_BLANK(parser->buffer))
3498	{
3499	/* Check for tab character that abuse intendation. */
3500
3501	if (leading_blanks && (int)parser->mark.column < indent
3502	&& IS_TAB(parser->buffer)) {
3503	yaml_parser_set_scanner_error(parser, "while scanning a plain scalar",
3504	start_mark, "found a tab character that violate intendation");
3505	goto error;
3506	}
3507
3508	/* Consume a space or a tab character. */
3509
3510	if (!leading_blanks) {
3511	if (!READ(parser, whitespaces)) goto error;
3512	}
3513	else {
3514	SKIP(parser);
3515	}
3516	}
3517	else
3518	{
3519	if (!CACHE(parser, 2)) goto error;
3520
3521	/* Check if it is a first line break. */
3522
3523	if (!leading_blanks)
3524	{
3525	CLEAR(parser, whitespaces);
3526	if (!READ_LINE(parser, leading_break)) goto error;
3527	leading_blanks = 1;
3528	}
3529	else
3530	{
3531	if (!READ_LINE(parser, trailing_breaks)) goto error;
3532	}
3533	}
3534	if (!CACHE(parser, 1)) goto error;
3535	}
3536
3537	/* Check intendation level. */
3538
3539	if (!parser->flow_level && (int)parser->mark.column < indent)
3540	break;
3541	}
3542
3543	/* Create a token. */
3544
3545	SCALAR_TOKEN_INIT(*token, string.start, string.pointer-string.start,
3546	YAML_PLAIN_SCALAR_STYLE, start_mark, end_mark);
3547
3548	/* Note that we change the 'simple_key_allowed' flag. */
3549
3550	if (leading_blanks) {
3551	parser->simple_key_allowed = 1;
3552	}
3553
3554	STRING_DEL(parser, leading_break);
3555	STRING_DEL(parser, trailing_breaks);
3556	STRING_DEL(parser, whitespaces);
3557
3558	return 1;
3559
3560	error:
3561	STRING_DEL(parser, string);
3562	STRING_DEL(parser, leading_break);
3563	STRING_DEL(parser, trailing_breaks);
3564	STRING_DEL(parser, whitespaces);
3565
3566	return 0;
3567	}
3568

Note: See TracBrowser for help on using the repository browser.

Download in other formats: