codekingpro/portable-devtools

1#
2# Copyright 2009 Facebook
3#
4# Licensed under the Apache License, Version 2.0 (the "License"); you may
5# not use this file except in compliance with the License. You may obtain
6# a copy of the License at
7#
8#     http://www.apache.org/licenses/LICENSE-2.0
9#
10# Unless required by applicable law or agreed to in writing, software
11# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
12# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13# License for the specific language governing permissions and limitations
14# under the License.
15
16"""A simple template system that compiles templates to Python code.
17
18Basic usage looks like::
19
20    t = template.Template("<html>{{ myvalue }}</html>")
21    print(t.generate(myvalue="XXX"))
22
23`Loader` is a class that loads templates from a root directory and caches
24the compiled templates::
25
26    loader = template.Loader("/home/btaylor")
27    print(loader.load("test.html").generate(myvalue="XXX"))
28
29We compile all templates to raw Python. Error-reporting is currently... uh,
30interesting. Syntax for the templates::
31
32    ### base.html
33    <html>
34      <head>
35        <title>{% block title %}Default title{% end %}</title>
36      </head>
37      <body>
38        <ul>
39          {% for student in students %}
40            {% block student %}
41              <li>{{ escape(student.name) }}</li>
42            {% end %}
43          {% end %}
44        </ul>
45      </body>
46    </html>
47
48    ### bold.html
49    {% extends "base.html" %}
50
51    {% block title %}A bolder title{% end %}
52
53    {% block student %}
54      <li><span style="bold">{{ escape(student.name) }}</span></li>
55    {% end %}
56
57Unlike most other template systems, we do not put any restrictions on the
58expressions you can include in your statements. ``if`` and ``for`` blocks get
59translated exactly into Python, so you can do complex expressions like::
60
61   {% for student in [p for p in people if p.student and p.age > 23] %}
62     <li>{{ escape(student.name) }}</li>
63   {% end %}
64
65Translating directly to Python means you can apply functions to expressions
66easily, like the ``escape()`` function in the examples above. You can pass
67functions in to your template just like any other variable
68(In a `.RequestHandler`, override `.RequestHandler.get_template_namespace`)::
69
70   ### Python code
71   def add(x, y):
72      return x + y
73   template.execute(add=add)
74
75   ### The template
76   {{ add(1, 2) }}
77
78We provide the functions `escape() <.xhtml_escape>`, `.url_escape()`,
79`.json_encode()`, and `.squeeze()` to all templates by default.
80
81Typical applications do not create `Template` or `Loader` instances by
82hand, but instead use the `~.RequestHandler.render` and
83`~.RequestHandler.render_string` methods of
84`tornado.web.RequestHandler`, which load templates automatically based
85on the ``template_path`` `.Application` setting.
86
87Variable names beginning with ``_tt_`` are reserved by the template
88system and should not be used by application code.
89
90Syntax Reference
91----------------
92
93Template expressions are surrounded by double curly braces: ``{{ ... }}``.
94The contents may be any python expression, which will be escaped according
95to the current autoescape setting and inserted into the output.  Other
96template directives use ``{% %}``.
97
98To comment out a section so that it is omitted from the output, surround it
99with ``{# ... #}``.
100
101
102To include a literal ``{{``, ``{%``, or ``{#`` in the output, escape them as
103``{{!``, ``{%!``, and ``{#!``, respectively.
104
105
106``{% apply *function* %}...{% end %}``
107    Applies a function to the output of all template code between ``apply``
108    and ``end``::
109
110        {% apply linkify %}{{name}} said: {{message}}{% end %}
111
112    Note that as an implementation detail apply blocks are implemented
113    as nested functions and thus may interact strangely with variables
114    set via ``{% set %}``, or the use of ``{% break %}`` or ``{% continue %}``
115    within loops.
116
117``{% autoescape *function* %}``
118    Sets the autoescape mode for the current file.  This does not affect
119    other files, even those referenced by ``{% include %}``.  Note that
120    autoescaping can also be configured globally, at the `.Application`
121    or `Loader`.::
122
123        {% autoescape xhtml_escape %}
124        {% autoescape None %}
125
126``{% block *name* %}...{% end %}``
127    Indicates a named, replaceable block for use with ``{% extends %}``.
128    Blocks in the parent template will be replaced with the contents of
129    the same-named block in a child template.::
130
131        <!-- base.html -->
132        <title>{% block title %}Default title{% end %}</title>
133
134        <!-- mypage.html -->
135        {% extends "base.html" %}
136        {% block title %}My page title{% end %}
137
138``{% comment ... %}``
139    A comment which will be removed from the template output.  Note that
140    there is no ``{% end %}`` tag; the comment goes from the word ``comment``
141    to the closing ``%}`` tag.
142
143``{% extends *filename* %}``
144    Inherit from another template.  Templates that use ``extends`` should
145    contain one or more ``block`` tags to replace content from the parent
146    template.  Anything in the child template not contained in a ``block``
147    tag will be ignored.  For an example, see the ``{% block %}`` tag.
148
149``{% for *var* in *expr* %}...{% end %}``
150    Same as the python ``for`` statement.  ``{% break %}`` and
151    ``{% continue %}`` may be used inside the loop.
152
153``{% from *x* import *y* %}``
154    Same as the python ``import`` statement.
155
156``{% if *condition* %}...{% elif *condition* %}...{% else %}...{% end %}``
157    Conditional statement - outputs the first section whose condition is
158    true.  (The ``elif`` and ``else`` sections are optional)
159
160``{% import *module* %}``
161    Same as the python ``import`` statement.
162
163``{% include *filename* %}``
164    Includes another template file.  The included file can see all the local
165    variables as if it were copied directly to the point of the ``include``
166    directive (the ``{% autoescape %}`` directive is an exception).
167    Alternately, ``{% module Template(filename, **kwargs) %}`` may be used
168    to include another template with an isolated namespace.
169
170``{% module *expr* %}``
171    Renders a `~tornado.web.UIModule`.  The output of the ``UIModule`` is
172    not escaped::
173
174        {% module Template("foo.html", arg=42) %}
175
176    ``UIModules`` are a feature of the `tornado.web.RequestHandler`
177    class (and specifically its ``render`` method) and will not work
178    when the template system is used on its own in other contexts.
179
180``{% raw *expr* %}``
181    Outputs the result of the given expression without autoescaping.
182
183``{% set *x* = *y* %}``
184    Sets a local variable.
185
186``{% try %}...{% except %}...{% else %}...{% finally %}...{% end %}``
187    Same as the python ``try`` statement.
188
189``{% while *condition* %}... {% end %}``
190    Same as the python ``while`` statement.  ``{% break %}`` and
191    ``{% continue %}`` may be used inside the loop.
192
193``{% whitespace *mode* %}``
194    Sets the whitespace mode for the remainder of the current file
195    (or until the next ``{% whitespace %}`` directive). See
196    `filter_whitespace` for available options. New in Tornado 4.3.
197"""
198
199import datetime
200from io import StringIO
201import linecache
202import os.path
203import posixpath
204import re
205import threading
206
207from tornado import escape
208from tornado.log import app_log
209from tornado.util import ObjectDict, exec_in, unicode_type
210
211from typing import Any, Union, Callable, List, Dict, Iterable, Optional, TextIO
212import typing
213
214if typing.TYPE_CHECKING:
215    from typing import Tuple, ContextManager  # noqa: F401
216
217_DEFAULT_AUTOESCAPE = "xhtml_escape"
218
219
220class _UnsetMarker:
221    pass
222
223
224_UNSET = _UnsetMarker()
225
226
227def filter_whitespace(mode: str, text: str) -> str:
228    """Transform whitespace in ``text`` according to ``mode``.
229
230    Available modes are:
231
232    * ``all``: Return all whitespace unmodified.
233    * ``single``: Collapse consecutive whitespace with a single whitespace
234      character, preserving newlines.
235    * ``oneline``: Collapse all runs of whitespace into a single space
236      character, removing all newlines in the process.
237
238    .. versionadded:: 4.3
239    """
240    if mode == "all":
241        return text
242    elif mode == "single":
243        text = re.sub(r"([\t ]+)", " ", text)
244        text = re.sub(r"(\s*\n\s*)", "\n", text)
245        return text
246    elif mode == "oneline":
247        return re.sub(r"(\s+)", " ", text)
248    else:
249        raise Exception("invalid whitespace mode %s" % mode)
250
251
252class Template:
253    """A compiled template.
254
255    We compile into Python from the given template_string. You can generate
256    the template from variables with generate().
257    """
258
259    # note that the constructor's signature is not extracted with
260    # autodoc because _UNSET looks like garbage.  When changing
261    # this signature update website/sphinx/template.rst too.
262    def __init__(
263        self,
264        template_string: Union[str, bytes],
265        name: str = "<string>",
266        loader: Optional["BaseLoader"] = None,
267        compress_whitespace: Union[bool, _UnsetMarker] = _UNSET,
268        autoescape: Optional[Union[str, _UnsetMarker]] = _UNSET,
269        whitespace: Optional[str] = None,
270    ) -> None:
271        """Construct a Template.
272
273        :arg str template_string: the contents of the template file.
274        :arg str name: the filename from which the template was loaded
275            (used for error message).
276        :arg tornado.template.BaseLoader loader: the `~tornado.template.BaseLoader` responsible
277            for this template, used to resolve ``{% include %}`` and ``{% extend %}`` directives.
278        :arg bool compress_whitespace: Deprecated since Tornado 4.3.
279            Equivalent to ``whitespace="single"`` if true and
280            ``whitespace="all"`` if false.
281        :arg str autoescape: The name of a function in the template
282            namespace, or ``None`` to disable escaping by default.
283        :arg str whitespace: A string specifying treatment of whitespace;
284            see `filter_whitespace` for options.
285
286        .. versionchanged:: 4.3
287           Added ``whitespace`` parameter; deprecated ``compress_whitespace``.
288        """
289        self.name = escape.native_str(name)
290
291        if compress_whitespace is not _UNSET:
292            # Convert deprecated compress_whitespace (bool) to whitespace (str).
293            if whitespace is not None:
294                raise Exception("cannot set both whitespace and compress_whitespace")
295            whitespace = "single" if compress_whitespace else "all"
296        if whitespace is None:
297            if loader and loader.whitespace:
298                whitespace = loader.whitespace
299            else:
300                # Whitespace defaults by filename.
301                if name.endswith(".html") or name.endswith(".js"):
302                    whitespace = "single"
303                else:
304                    whitespace = "all"
305        # Validate the whitespace setting.
306        assert whitespace is not None
307        filter_whitespace(whitespace, "")
308
309        if not isinstance(autoescape, _UnsetMarker):
310            self.autoescape = autoescape  # type: Optional[str]
311        elif loader:
312            self.autoescape = loader.autoescape
313        else:
314            self.autoescape = _DEFAULT_AUTOESCAPE
315
316        self.namespace = loader.namespace if loader else {}
317        reader = _TemplateReader(name, escape.native_str(template_string), whitespace)
318        self.file = _File(self, _parse(reader, self))
319        self.code = self._generate_python(loader)
320        self.loader = loader
321        try:
322            # Under python2.5, the fake filename used here must match
323            # the module name used in __name__ below.
324            # The dont_inherit flag prevents template.py's future imports
325            # from being applied to the generated code.
326            self.compiled = compile(
327                escape.to_unicode(self.code),
328                "%s.generated.py" % self.name.replace(".", "_"),
329                "exec",
330                dont_inherit=True,
331            )
332        except Exception:
333            formatted_code = _format_code(self.code).rstrip()
334            app_log.error("%s code:\n%s", self.name, formatted_code)
335            raise
336
337    def generate(self, **kwargs: Any) -> bytes:
338        """Generate this template with the given arguments."""
339        namespace = {
340            "escape": escape.xhtml_escape,
341            "xhtml_escape": escape.xhtml_escape,
342            "url_escape": escape.url_escape,
343            "json_encode": escape.json_encode,
344            "squeeze": escape.squeeze,
345            "linkify": escape.linkify,
346            "datetime": datetime,
347            "_tt_utf8": escape.utf8,  # for internal use
348            "_tt_string_types": (unicode_type, bytes),
349            # __name__ and __loader__ allow the traceback mechanism to find
350            # the generated source code.
351            "__name__": self.name.replace(".", "_"),
352            "__loader__": ObjectDict(get_source=lambda name: self.code),
353        }
354        namespace.update(self.namespace)
355        namespace.update(kwargs)
356        exec_in(self.compiled, namespace)
357        execute = typing.cast(Callable[[], bytes], namespace["_tt_execute"])
358        # Clear the traceback module's cache of source data now that
359        # we've generated a new template (mainly for this module's
360        # unittests, where different tests reuse the same name).
361        linecache.clearcache()
362        return execute()
363
364    def _generate_python(self, loader: Optional["BaseLoader"]) -> str:
365        buffer = StringIO()
366        try:
367            # named_blocks maps from names to _NamedBlock objects
368            named_blocks = {}  # type: Dict[str, _NamedBlock]
369            ancestors = self._get_ancestors(loader)
370            ancestors.reverse()
371            for ancestor in ancestors:
372                ancestor.find_named_blocks(loader, named_blocks)
373            writer = _CodeWriter(buffer, named_blocks, loader, ancestors[0].template)
374            ancestors[0].generate(writer)
375            return buffer.getvalue()
376        finally:
377            buffer.close()
378
379    def _get_ancestors(self, loader: Optional["BaseLoader"]) -> List["_File"]:
380        ancestors = [self.file]
381        for chunk in self.file.body.chunks:
382            if isinstance(chunk, _ExtendsBlock):
383                if not loader:
384                    raise ParseError(
385                        "{% extends %} block found, but no " "template loader"
386                    )
387                template = loader.load(chunk.name, self.name)
388                ancestors.extend(template._get_ancestors(loader))
389        return ancestors
390
391
392class BaseLoader:
393    """Base class for template loaders.
394
395    You must use a template loader to use template constructs like
396    ``{% extends %}`` and ``{% include %}``. The loader caches all
397    templates after they are loaded the first time.
398    """
399
400    def __init__(
401        self,
402        autoescape: Optional[str] = _DEFAULT_AUTOESCAPE,
403        namespace: Optional[Dict[str, Any]] = None,
404        whitespace: Optional[str] = None,
405    ) -> None:
406        """Construct a template loader.
407
408        :arg str autoescape: The name of a function in the template
409            namespace, such as "xhtml_escape", or ``None`` to disable
410            autoescaping by default.
411        :arg dict namespace: A dictionary to be added to the default template
412            namespace, or ``None``.
413        :arg str whitespace: A string specifying default behavior for
414            whitespace in templates; see `filter_whitespace` for options.
415            Default is "single" for files ending in ".html" and ".js" and
416            "all" for other files.
417
418        .. versionchanged:: 4.3
419           Added ``whitespace`` parameter.
420        """
421        self.autoescape = autoescape
422        self.namespace = namespace or {}
423        self.whitespace = whitespace
424        self.templates = {}  # type: Dict[str, Template]
425        # self.lock protects self.templates.  It's a reentrant lock
426        # because templates may load other templates via `include` or
427        # `extends`.  Note that thanks to the GIL this code would be safe
428        # even without the lock, but could lead to wasted work as multiple
429        # threads tried to compile the same template simultaneously.
430        self.lock = threading.RLock()
431
432    def reset(self) -> None:
433        """Resets the cache of compiled templates."""
434        with self.lock:
435            self.templates = {}
436
437    def resolve_path(self, name: str, parent_path: Optional[str] = None) -> str:
438        """Converts a possibly-relative path to absolute (used internally)."""
439        raise NotImplementedError()
440
441    def load(self, name: str, parent_path: Optional[str] = None) -> Template:
442        """Loads a template."""
443        name = self.resolve_path(name, parent_path=parent_path)
444        with self.lock:
445            if name not in self.templates:
446                self.templates[name] = self._create_template(name)
447            return self.templates[name]
448
449    def _create_template(self, name: str) -> Template:
450        raise NotImplementedError()
451
452
453class Loader(BaseLoader):
454    """A template loader that loads from a single root directory."""
455
456    def __init__(self, root_directory: str, **kwargs: Any) -> None:
457        super().__init__(**kwargs)
458        self.root = os.path.abspath(root_directory)
459
460    def resolve_path(self, name: str, parent_path: Optional[str] = None) -> str:
461        if (
462            parent_path
463            and not parent_path.startswith("<")
464            and not parent_path.startswith("/")
465            and not name.startswith("/")
466        ):
467            current_path = os.path.join(self.root, parent_path)
468            file_dir = os.path.dirname(os.path.abspath(current_path))
469            relative_path = os.path.abspath(os.path.join(file_dir, name))
470            if relative_path.startswith(self.root):
471                name = relative_path[len(self.root) + 1 :]
472        return name
473
474    def _create_template(self, name: str) -> Template:
475        path = os.path.join(self.root, name)
476        with open(path, "rb") as f:
477            template = Template(f.read(), name=name, loader=self)
478            return template
479
480
481class DictLoader(BaseLoader):
482    """A template loader that loads from a dictionary."""
483
484    def __init__(self, dict: Dict[str, str], **kwargs: Any) -> None:
485        super().__init__(**kwargs)
486        self.dict = dict
487
488    def resolve_path(self, name: str, parent_path: Optional[str] = None) -> str:
489        if (
490            parent_path
491            and not parent_path.startswith("<")
492            and not parent_path.startswith("/")
493            and not name.startswith("/")
494        ):
495            file_dir = posixpath.dirname(parent_path)
496            name = posixpath.normpath(posixpath.join(file_dir, name))
497        return name
498
499    def _create_template(self, name: str) -> Template:
500        return Template(self.dict[name], name=name, loader=self)
501
502
503class _Node:
504    def each_child(self) -> Iterable["_Node"]:
505        return ()
506
507    def generate(self, writer: "_CodeWriter") -> None:
508        raise NotImplementedError()
509
510    def find_named_blocks(
511        self, loader: Optional[BaseLoader], named_blocks: Dict[str, "_NamedBlock"]
512    ) -> None:
513        for child in self.each_child():
514            child.find_named_blocks(loader, named_blocks)
515
516
517class _File(_Node):
518    def __init__(self, template: Template, body: "_ChunkList") -> None:
519        self.template = template
520        self.body = body
521        self.line = 0
522
523    def generate(self, writer: "_CodeWriter") -> None:
524        writer.write_line("def _tt_execute():", self.line)
525        with writer.indent():
526            writer.write_line("_tt_buffer = []", self.line)
527            writer.write_line("_tt_append = _tt_buffer.append", self.line)
528            self.body.generate(writer)
529            writer.write_line("return _tt_utf8('').join(_tt_buffer)", self.line)
530
531    def each_child(self) -> Iterable["_Node"]:
532        return (self.body,)
533
534
535class _ChunkList(_Node):
536    def __init__(self, chunks: List[_Node]) -> None:
537        self.chunks = chunks
538
539    def generate(self, writer: "_CodeWriter") -> None:
540        for chunk in self.chunks:
541            chunk.generate(writer)
542
543    def each_child(self) -> Iterable["_Node"]:
544        return self.chunks
545
546
547class _NamedBlock(_Node):
548    def __init__(self, name: str, body: _Node, template: Template, line: int) -> None:
549        self.name = name
550        self.body = body
551        self.template = template
552        self.line = line
553
554    def each_child(self) -> Iterable["_Node"]:
555        return (self.body,)
556
557    def generate(self, writer: "_CodeWriter") -> None:
558        block = writer.named_blocks[self.name]
559        with writer.include(block.template, self.line):
560            block.body.generate(writer)
561
562    def find_named_blocks(
563        self, loader: Optional[BaseLoader], named_blocks: Dict[str, "_NamedBlock"]
564    ) -> None:
565        named_blocks[self.name] = self
566        _Node.find_named_blocks(self, loader, named_blocks)
567
568
569class _ExtendsBlock(_Node):
570    def __init__(self, name: str) -> None:
571        self.name = name
572
573
574class _IncludeBlock(_Node):
575    def __init__(self, name: str, reader: "_TemplateReader", line: int) -> None:
576        self.name = name
577        self.template_name = reader.name
578        self.line = line
579
580    def find_named_blocks(
581        self, loader: Optional[BaseLoader], named_blocks: Dict[str, _NamedBlock]
582    ) -> None:
583        assert loader is not None
584        included = loader.load(self.name, self.template_name)
585        included.file.find_named_blocks(loader, named_blocks)
586
587    def generate(self, writer: "_CodeWriter") -> None:
588        assert writer.loader is not None
589        included = writer.loader.load(self.name, self.template_name)
590        with writer.include(included, self.line):
591            included.file.body.generate(writer)
592
593
594class _ApplyBlock(_Node):
595    def __init__(self, method: str, line: int, body: _Node) -> None:
596        self.method = method
597        self.line = line
598        self.body = body
599
600    def each_child(self) -> Iterable["_Node"]:
601        return (self.body,)
602
603    def generate(self, writer: "_CodeWriter") -> None:
604        method_name = "_tt_apply%d" % writer.apply_counter
605        writer.apply_counter += 1
606        writer.write_line("def %s():" % method_name, self.line)
607        with writer.indent():
608            writer.write_line("_tt_buffer = []", self.line)
609            writer.write_line("_tt_append = _tt_buffer.append", self.line)
610            self.body.generate(writer)
611            writer.write_line("return _tt_utf8('').join(_tt_buffer)", self.line)
612        writer.write_line(
613            f"_tt_append(_tt_utf8({self.method}({method_name}())))", self.line
614        )
615
616
617class _ControlBlock(_Node):
618    def __init__(self, statement: str, line: int, body: _Node) -> None:
619        self.statement = statement
620        self.line = line
621        self.body = body
622
623    def each_child(self) -> Iterable[_Node]:
624        return (self.body,)
625
626    def generate(self, writer: "_CodeWriter") -> None:
627        writer.write_line("%s:" % self.statement, self.line)
628        with writer.indent():
629            self.body.generate(writer)
630            # Just in case the body was empty
631            writer.write_line("pass", self.line)
632
633
634class _IntermediateControlBlock(_Node):
635    def __init__(self, statement: str, line: int) -> None:
636        self.statement = statement
637        self.line = line
638
639    def generate(self, writer: "_CodeWriter") -> None:
640        # In case the previous block was empty
641        writer.write_line("pass", self.line)
642        writer.write_line("%s:" % self.statement, self.line, writer.indent_size() - 1)
643
644
645class _Statement(_Node):
646    def __init__(self, statement: str, line: int) -> None:
647        self.statement = statement
648        self.line = line
649
650    def generate(self, writer: "_CodeWriter") -> None:
651        writer.write_line(self.statement, self.line)
652
653
654class _Expression(_Node):
655    def __init__(self, expression: str, line: int, raw: bool = False) -> None:
656        self.expression = expression
657        self.line = line
658        self.raw = raw
659
660    def generate(self, writer: "_CodeWriter") -> None:
661        writer.write_line("_tt_tmp = %s" % self.expression, self.line)
662        writer.write_line(
663            "if isinstance(_tt_tmp, _tt_string_types):" " _tt_tmp = _tt_utf8(_tt_tmp)",
664            self.line,
665        )
666        writer.write_line("else: _tt_tmp = _tt_utf8(str(_tt_tmp))", self.line)
667        if not self.raw and writer.current_template.autoescape is not None:
668            # In python3 functions like xhtml_escape return unicode,
669            # so we have to convert to utf8 again.
670            writer.write_line(
671                "_tt_tmp = _tt_utf8(%s(_tt_tmp))" % writer.current_template.autoescape,
672                self.line,
673            )
674        writer.write_line("_tt_append(_tt_tmp)", self.line)
675
676
677class _Module(_Expression):
678    def __init__(self, expression: str, line: int) -> None:
679        super().__init__("_tt_modules." + expression, line, raw=True)
680
681
682class _Text(_Node):
683    def __init__(self, value: str, line: int, whitespace: str) -> None:
684        self.value = value
685        self.line = line
686        self.whitespace = whitespace
687
688    def generate(self, writer: "_CodeWriter") -> None:
689        value = self.value
690
691        # Compress whitespace if requested, with a crude heuristic to avoid
692        # altering preformatted whitespace.
693        if "<pre>" not in value:
694            value = filter_whitespace(self.whitespace, value)
695
696        if value:
697            writer.write_line("_tt_append(%r)" % escape.utf8(value), self.line)
698
699
700class ParseError(Exception):
701    """Raised for template syntax errors.
702
703    ``ParseError`` instances have ``filename`` and ``lineno`` attributes
704    indicating the position of the error.
705
706    .. versionchanged:: 4.3
707       Added ``filename`` and ``lineno`` attributes.
708    """
709
710    def __init__(
711        self, message: str, filename: Optional[str] = None, lineno: int = 0
712    ) -> None:
713        self.message = message
714        # The names "filename" and "lineno" are chosen for consistency
715        # with python SyntaxError.
716        self.filename = filename
717        self.lineno = lineno
718
719    def __str__(self) -> str:
720        return "%s at %s:%d" % (self.message, self.filename, self.lineno)
721
722
723class _CodeWriter:
724    def __init__(
725        self,
726        file: TextIO,
727        named_blocks: Dict[str, _NamedBlock],
728        loader: Optional[BaseLoader],
729        current_template: Template,
730    ) -> None:
731        self.file = file
732        self.named_blocks = named_blocks
733        self.loader = loader
734        self.current_template = current_template
735        self.apply_counter = 0
736        self.include_stack = []  # type: List[Tuple[Template, int]]
737        self._indent = 0
738
739    def indent_size(self) -> int:
740        return self._indent
741
742    def indent(self) -> "ContextManager":
743        class Indenter:
744            def __enter__(_) -> "_CodeWriter":
745                self._indent += 1
746                return self
747
748            def __exit__(_, *args: Any) -> None:
749                assert self._indent > 0
750                self._indent -= 1
751
752        return Indenter()
753
754    def include(self, template: Template, line: int) -> "ContextManager":
755        self.include_stack.append((self.current_template, line))
756        self.current_template = template
757
758        class IncludeTemplate:
759            def __enter__(_) -> "_CodeWriter":
760                return self
761
762            def __exit__(_, *args: Any) -> None:
763                self.current_template = self.include_stack.pop()[0]
764
765        return IncludeTemplate()
766
767    def write_line(
768        self, line: str, line_number: int, indent: Optional[int] = None
769    ) -> None:
770        if indent is None:
771            indent = self._indent
772        line_comment = "  # %s:%d" % (self.current_template.name, line_number)
773        if self.include_stack:
774            ancestors = [
775                "%s:%d" % (tmpl.name, lineno) for (tmpl, lineno) in self.include_stack
776            ]
777            line_comment += " (via %s)" % ", ".join(reversed(ancestors))
778        print("    " * indent + line + line_comment, file=self.file)
779
780
781class _TemplateReader:
782    def __init__(self, name: str, text: str, whitespace: str) -> None:
783        self.name = name
784        self.text = text
785        self.whitespace = whitespace
786        self.line = 1
787        self.pos = 0
788
789    def find(self, needle: str, start: int = 0, end: Optional[int] = None) -> int:
790        assert start >= 0, start
791        pos = self.pos
792        start += pos
793        if end is None:
794            index = self.text.find(needle, start)
795        else:
796            end += pos
797            assert end >= start
798            index = self.text.find(needle, start, end)
799        if index != -1:
800            index -= pos
801        return index
802
803    def consume(self, count: Optional[int] = None) -> str:
804        if count is None:
805            count = len(self.text) - self.pos
806        newpos = self.pos + count
807        self.line += self.text.count("\n", self.pos, newpos)
808        s = self.text[self.pos : newpos]
809        self.pos = newpos
810        return s
811
812    def remaining(self) -> int:
813        return len(self.text) - self.pos
814
815    def __len__(self) -> int:
816        return self.remaining()
817
818    def __getitem__(self, key: Union[int, slice]) -> str:
819        if isinstance(key, slice):
820            size = len(self)
821            start, stop, step = key.indices(size)
822            if start is None:
823                start = self.pos
824            else:
825                start += self.pos
826            if stop is not None:
827                stop += self.pos
828            return self.text[slice(start, stop, step)]
829        elif key < 0:
830            return self.text[key]
831        else:
832            return self.text[self.pos + key]
833
834    def __str__(self) -> str:
835        return self.text[self.pos :]
836
837    def raise_parse_error(self, msg: str) -> None:
838        raise ParseError(msg, self.name, self.line)
839
840
841def _format_code(code: str) -> str:
842    lines = code.splitlines()
843    format = "%%%dd  %%s\n" % len(repr(len(lines) + 1))
844    return "".join([format % (i + 1, line) for (i, line) in enumerate(lines)])
845
846
847def _parse(
848    reader: _TemplateReader,
849    template: Template,
850    in_block: Optional[str] = None,
851    in_loop: Optional[str] = None,
852) -> _ChunkList:
853    body = _ChunkList([])
854    while True:
855        # Find next template directive
856        curly = 0
857        while True:
858            curly = reader.find("{", curly)
859            if curly == -1 or curly + 1 == reader.remaining():
860                # EOF
861                if in_block:
862                    reader.raise_parse_error(
863                        "Missing {%% end %%} block for %s" % in_block
864                    )
865                body.chunks.append(
866                    _Text(reader.consume(), reader.line, reader.whitespace)
867                )
868                return body
869            # If the first curly brace is not the start of a special token,
870            # start searching from the character after it
871            if reader[curly + 1] not in ("{", "%", "#"):
872                curly += 1
873                continue
874            # When there are more than 2 curlies in a row, use the
875            # innermost ones.  This is useful when generating languages
876            # like latex where curlies are also meaningful
877            if (
878                curly + 2 < reader.remaining()
879                and reader[curly + 1] == "{"
880                and reader[curly + 2] == "{"
881            ):
882                curly += 1
883                continue
884            break
885
886        # Append any text before the special token
887        if curly > 0:
888            cons = reader.consume(curly)
889            body.chunks.append(_Text(cons, reader.line, reader.whitespace))
890
891        start_brace = reader.consume(2)
892        line = reader.line
893
894        # Template directives may be escaped as "{{!" or "{%!".
895        # In this case output the braces and consume the "!".
896        # This is especially useful in conjunction with jquery templates,
897        # which also use double braces.
898        if reader.remaining() and reader[0] == "!":
899            reader.consume(1)
900            body.chunks.append(_Text(start_brace, line, reader.whitespace))
901            continue
902
903        # Comment
904        if start_brace == "{#":
905            end = reader.find("#}")
906            if end == -1:
907                reader.raise_parse_error("Missing end comment #}")
908            contents = reader.consume(end).strip()
909            reader.consume(2)
910            continue
911
912        # Expression
913        if start_brace == "{{":
914            end = reader.find("}}")
915            if end == -1:
916                reader.raise_parse_error("Missing end expression }}")
917            contents = reader.consume(end).strip()
918            reader.consume(2)
919            if not contents:
920                reader.raise_parse_error("Empty expression")
921            body.chunks.append(_Expression(contents, line))
922            continue
923
924        # Block
925        assert start_brace == "{%", start_brace
926        end = reader.find("%}")
927        if end == -1:
928            reader.raise_parse_error("Missing end block %}")
929        contents = reader.consume(end).strip()
930        reader.consume(2)
931        if not contents:
932            reader.raise_parse_error("Empty block tag ({% %})")
933
934        operator, space, suffix = contents.partition(" ")
935        suffix = suffix.strip()
936
937        # Intermediate ("else", "elif", etc) blocks
938        intermediate_blocks = {
939            "else": {"if", "for", "while", "try"},
940            "elif": {"if"},
941            "except": {"try"},
942            "finally": {"try"},
943        }
944        allowed_parents = intermediate_blocks.get(operator)
945        if allowed_parents is not None:
946            if not in_block:
947                reader.raise_parse_error(f"{operator} outside {allowed_parents} block")
948            if in_block not in allowed_parents:
949                reader.raise_parse_error(
950                    f"{operator} block cannot be attached to {in_block} block"
951                )
952            body.chunks.append(_IntermediateControlBlock(contents, line))
953            continue
954
955        # End tag
956        elif operator == "end":
957            if not in_block:
958                reader.raise_parse_error("Extra {% end %} block")
959            return body
960
961        elif operator in (
962            "extends",
963            "include",
964            "set",
965            "import",
966            "from",
967            "comment",
968            "autoescape",
969            "whitespace",
970            "raw",
971            "module",
972        ):
973            if operator == "comment":
974                continue
975            if operator == "extends":
976                suffix = suffix.strip('"').strip("'")
977                if not suffix:
978                    reader.raise_parse_error("extends missing file path")
979                block = _ExtendsBlock(suffix)  # type: _Node
980            elif operator in ("import", "from"):
981                if not suffix:
982                    reader.raise_parse_error("import missing statement")
983                block = _Statement(contents, line)
984            elif operator == "include":
985                suffix = suffix.strip('"').strip("'")
986                if not suffix:
987                    reader.raise_parse_error("include missing file path")
988                block = _IncludeBlock(suffix, reader, line)
989            elif operator == "set":
990                if not suffix:
991                    reader.raise_parse_error("set missing statement")
992                block = _Statement(suffix, line)
993            elif operator == "autoescape":
994                fn = suffix.strip()  # type: Optional[str]
995                if fn == "None":
996                    fn = None
997                template.autoescape = fn
998                continue
999            elif operator == "whitespace":
1000                mode = suffix.strip()
1001                # Validate the selected mode
1002                filter_whitespace(mode, "")
1003                reader.whitespace = mode
1004                continue
1005            elif operator == "raw":
1006                block = _Expression(suffix, line, raw=True)
1007            elif operator == "module":
1008                block = _Module(suffix, line)
1009            body.chunks.append(block)
1010            continue
1011
1012        elif operator in ("apply", "block", "try", "if", "for", "while"):
1013            # parse inner body recursively
1014            if operator in ("for", "while"):
1015                block_body = _parse(reader, template, operator, operator)
1016            elif operator == "apply":
1017                # apply creates a nested function so syntactically it's not
1018                # in the loop.
1019                block_body = _parse(reader, template, operator, None)
1020            else:
1021                block_body = _parse(reader, template, operator, in_loop)
1022
1023            if operator == "apply":
1024                if not suffix:
1025                    reader.raise_parse_error("apply missing method name")
1026                block = _ApplyBlock(suffix, line, block_body)
1027            elif operator == "block":
1028                if not suffix:
1029                    reader.raise_parse_error("block missing name")
1030                block = _NamedBlock(suffix, block_body, template, line)
1031            else:
1032                block = _ControlBlock(contents, line, block_body)
1033            body.chunks.append(block)
1034            continue
1035
1036        elif operator in ("break", "continue"):
1037            if not in_loop:
1038                reader.raise_parse_error(
1039                    "{} outside {} block".format(operator, {"for", "while"})
1040                )
1041            body.chunks.append(_Statement(contents, line))
1042            continue
1043
1044        else:
1045            reader.raise_parse_error("unknown operator: %r" % operator)
1046