codekingpro/portable-devtools

1"""2This module provides the API for ``requests_toolbelt.threaded``.3 4The module provides a clean and simple API for making requests via a thread5pool. The thread pool will use sessions for increased performance.6 7A simple use-case is:8 9.. code-block:: python10 11    from requests_toolbelt import threaded12 13    urls_to_get = [{14        'url': 'https://api.github.com/users/sigmavirus24',15        'method': 'GET',16    }, {17        'url': 'https://api.github.com/repos/requests/toolbelt',18        'method': 'GET',19    }, {20        'url': 'https://google.com',21        'method': 'GET',22    }]23    responses, errors = threaded.map(urls_to_get)24 25By default, the threaded submodule will detect the number of CPUs your26computer has and use that if no other number of processes is selected. To27change this, always use the keyword argument ``num_processes``. Using the28above example, we would expand it like so:29 30.. code-block:: python31 32    responses, errors = threaded.map(urls_to_get, num_processes=10)33 34You can also customize how a :class:`requests.Session` is initialized by35creating a callback function:36 37.. code-block:: python38 39    from requests_toolbelt import user_agent40 41    def initialize_session(session):42        session.headers['User-Agent'] = user_agent('my-scraper', '0.1')43        session.headers['Accept'] = 'application/json'44 45    responses, errors = threaded.map(urls_to_get,46                                     initializer=initialize_session)47 48.. autofunction:: requests_toolbelt.threaded.map49 50Inspiration is blatantly drawn from the standard library's multiprocessing51library. See the following references:52 53- multiprocessing's `pool source`_54 55- map and map_async `inspiration`_56 57.. _pool source:58    https://hg.python.org/cpython/file/8ef4f75a8018/Lib/multiprocessing/pool.py59.. _inspiration:60    https://hg.python.org/cpython/file/8ef4f75a8018/Lib/multiprocessing/pool.py#l34061"""62from . import pool63from .._compat import queue64 65 66def map(requests, **kwargs):67    r"""Simple interface to the threaded Pool object.68 69    This function takes a list of dictionaries representing requests to make70    using Sessions in threads and returns a tuple where the first item is71    a generator of successful responses and the second is a generator of72    exceptions.73 74    :param list requests:75        Collection of dictionaries representing requests to make with the Pool76        object.77    :param \*\*kwargs:78        Keyword arguments that are passed to the79        :class:`~requests_toolbelt.threaded.pool.Pool` object.80    :returns: Tuple of responses and exceptions from the pool81    :rtype: (:class:`~requests_toolbelt.threaded.pool.ThreadResponse`,82        :class:`~requests_toolbelt.threaded.pool.ThreadException`)83    """84    if not (requests and all(isinstance(r, dict) for r in requests)):85        raise ValueError('map expects a list of dictionaries.')86 87    # Build our queue of requests88    job_queue = queue.Queue()89    for request in requests:90        job_queue.put(request)91 92    # Ensure the user doesn't try to pass their own job_queue93    kwargs['job_queue'] = job_queue94 95    threadpool = pool.Pool(**kwargs)96    threadpool.join_all()97    return threadpool.responses(), threadpool.exceptions()98