Как подключить куки в запрос ?

Discussion in 'Песочница' started by MrCable, 13 Feb 2016.

Page 1 of 2
  1. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Пытаюсь попасть на сайт скриптом, но в ответ Connection:close и Internal server error.
    <code>
    function curl($url){
    $headers = array(
    'GET ' . $url . ' HTTP/1.0',
    'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0',
    'Accept: text/html,
    application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3',
    'Accept-Encoding: gzip, deflate',
    'Connection: keep-alive'
    );



    $ch = curl_init();
    curl_setopt ($ch , CURLOPT_URL , $url );
    curl_setopt ($ch , CURLOPT_USERAGENT , "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0");
    curl_setopt ($ch, CURLOPT_HEADER, 1);
    curl_setopt ($ch, CURLOPT_REFERER, 'https://www.google.com.ua');
    curl_setopt ($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt ($ch, CURLOPT_COOKIEFILE, "cookie.txt");
    curl_setopt ($ch, CURLOPT_COOKIEJAR, "cookie.txt");
    curl_setopt ($ch , CURLOPT_RETURNTRANSFER , 1 );
    $file = curl_exec($ch);
    return $file;


    }
    <code>
    Сам запрос в Фиддер 4
    <code>
    GET / HTTP/1.1
    Host: it-weekly.ru
    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0
    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
    Accept-Encoding: gzip, deflate
    Cookie: BITRIX_SM_GUEST_ID=5887196; BITRIX_SM_LAST_VISIT=13.02.2016+16%3A16%3A25; BITRIX_SM_LAST_ADV=5_Y; __utma=24257250.1849781716.1455134909.1455351502.1455369036.6; __utmz=24257250.1455351502.5.5.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _ym_uid=1455134909594827418; BX_USER_ID=7f7ba1be030d47aea8577d104e129a4a; resolution=800; __utmc=24257250; PHPSESSID=c72285cf06fcd6764af901ebe891b8ed
    Connection: keep-alive

    <code>
    Декодированая версия
    <code>
    HTTP/1.1 200 OK
    Server: nginx/0.8.53
    Date: Sat, 13 Feb 2016 20:04:57 GMT
    Content-Type: text/html; charset=UTF-8
    Connection: keep-alive
    X-Powered-By: PHP/5.3.18
    P3P: policyref="/bitrix/p3p.xml", CP="NON DSP COR CUR ADM DEV PSA PSD OUR UNR BUS UNI COM NAV INT DEM STA"
    X-Powered-CMS: Bitrix Site Manager (19b839d1031a99892d4b0429297bad16)
    Expires: Thu, 19 Nov 1981 08:52:00 GMT
    Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    Pragma: no-cache
    X-Frame-Options: SAMEORIGIN
    Set-Cookie: PHPSESSID=24fa65693270aaec2831489b3897e176; path=/; domain=it-weekly.ru; HttpOnly
    Set-Cookie: BITRIX_SM_GUEST_ID=5887196; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru
    Set-Cookie: BITRIX_SM_LAST_VISIT=13.02.2016+23%3A04%3A53; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru
    Content-Length: 51223

    <code>
     
    #1 MrCable, 13 Feb 2016
  2. rct

    rct Active Member

    Joined:
    13 Jun 2015
    Messages:
    359
    Likes Received:
    107
    Reputations:
    7
    Code:
    import requests

hdrs = {
    "User-Agent" : "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0",
}

cks = {
    "param": "value",
    "param1": "value",
}

r = requests.get(url, headers=hdrs, cookies=cks)
     
    #2 rct, 14 Feb 2016
  3. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Это на питоне ? А есть вариант на пхп, или он очень замороченный ?
    П.С. Нубский вопрос, где теги форума глянуть ?
     
    #3 MrCable, 14 Feb 2016
  4. rct

    rct Active Member

    Joined:
    13 Jun 2015
    Messages:
    359
    Likes Received:
    107
    Reputations:
    7
    Да, питон. На пхп я не умею)
    Теги тут
    [​IMG]
     
    #4 rct, 14 Feb 2016
  5. Waki

    Waki Member

    Joined:
    9 Oct 2015
    Messages:
    54
    Likes Received:
    30
    Reputations:
    10
    docstore.mik.ua/orelly/webprog/pcook/ch11_04.htm
    почитай тут
     
    #5 Waki, 14 Feb 2016
  6. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    PHP:
    curl_setopt($c, CURLOPT_COOKIE, 'PHPSESSID=24fa65693270aaec2831489b3897e176; path=/; domain=it-weekly.ru; HttpOnly');
    curl_setopt($c, CURLOPT_COOKIE, 'BITRIX_SM_GUEST_ID=5887196; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru');
    curl_setopt($c, CURLOPT_COOKIE, 'BITRIX_SM_LAST_VISIT=13.02.2016+23%3A04%3A53; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru');
    Как я понял мне необходимо указать эти куки, данным образом ? Но разве PHPSESSID не должен быть разным ? да и остальные параметры скорей всего тоже меняются?!

    PHP:
    Set-Cookie: PHPSESSID=24fa65693270aaec2831489b3897e176; path=/; domain=it-weekly.ru; HttpOnly
    Set    -Cookie: BITRIX_SM_GUEST_ID=5887196; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru
    Set    -Cookie: BITRIX_SM_LAST_VISIT=13.02.2016+23%3A04%3A53; expires=Tue, 07-Feb-2017 20:04:53 GMT; path=/; domain=it-weekly.ru
     
    #6 MrCable, 14 Feb 2016
  7. Waki

    Waki Member

    Joined:
    9 Oct 2015
    Messages:
    54
    Likes Received:
    30
    Reputations:
    10
    Используй пример с CURLOPT_COOKIEJAR и CURLOPT_COOKIEFILE
     
    #7 Waki, 14 Feb 2016
  8. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Пробую следующий код проблема осталась, запрос не доходит, или закрывают в ответ на запрос 500 код.
    PHP:
    function curl($url){
            $cookie_jar = tempnam('/tmp','cookie');
       
            $headers = array(
            'GET ' . $url . ' HTTP/1.0',
            'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0',
            'Accept: text/html,
                      application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'    ,
            'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3',
            'Accept-Encoding: gzip, deflate',
            'Connection: keep-alive'
    );



            $ch = curl_init();
    curl_setopt ($ch , CURLOPT_URL , $url );
    curl_setopt ($ch , CURLOPT_USERAGENT , "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0");
    curl_setopt ($ch, CURLOPT_HEADER, 1);
    curl_setopt ($ch, CURLOPT_REFERER, 'https://www.google.com.ua');
    curl_setopt ($ch, CURLOPT_HTTPHEADER, $headers);
    //curl_setopt ($ch, CURLOPT_COOKIEFILE, "$cookie_jar");
    curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie_jar);
    curl_setopt ($ch , CURLOPT_RETURNTRANSFER , 1 );
    $file = curl_exec($ch);
            // remove the cookie jar
    unlink($cookie_jar) or die("Can't unlink $cookie_jar");
        return     $file;
      
       
    }
     
    #8 MrCable, 14 Feb 2016
  9. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    PHP:
        'Accept: text/html,
                      application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'    ,
    а этот перенос нормой в php считаеться?)
    для меня бы это был уже совсем другой хедер.. как в php хз
     
    #9 lifescore, 14 Feb 2016
  10. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    в любом случае оно не пашет, а принятие скорей всего строкой
     
    #10 MrCable, 14 Feb 2016
  11. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    скинь фидлер лог
    с 500 ошибкой
     
    #11 lifescore, 14 Feb 2016
  12. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Code:
    GET /news/script.php HTTP/1.1
Host: test.getfreebits.ru
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cache-Control: max-age=0
    ответ
    Code:
    -= Fiddler Event Log =-
See http://fiddler2.com/r/?FiddlerLog for details.

22:11:17:6920 Fiddler Running...
22:11:17:6930 !WARNING: System proxy was configured to use a file-protocol sourced script (C:\Program Files\i2p\scripts\i2pProxy.pac). Proxy scripts delivered by the file protocol are not supported by many clients. Please see http://blogs.msdn.com/b/ieinternals/archive/2013/10/11/web-proxy-configuration-and-ie11-changes.aspx for more information.
    Code:
    HTTP/1.1 500 Internal Server Error
Date: Sun, 14 Feb 2016 20:11:35 GMT
Server: Apache
Content-Length: 661
Connection: close
Content-Type: text/html; charset=iso-8859-1
     
    #12 MrCable, 14 Feb 2016
  13. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    о господи...

    [​IMG]

    С начало GET запрос отправляй, получай куки гостя


    Code:
    GET http://it-weekly.ru/personal/ HTTP/1.1
Host: it-weekly.ru
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36
DNT: 1
Accept-Encoding: gzip, deflate, sdch
Accept-Language: ru,en;q=0.8
Cookie: _ym_uid=1455478095228020382; BX_USER_ID=f9d2a4228ad5ae9bd08b29889bb3aaf6; _ym_isad=0; PHPSESSID=704733ef09cefae931bce5b9bc1a9d1f; resolution=2560; __utma=24257250.2079285695.1455478095.1455478095.1455478095.1; __utmc=24257250; __utmz=24257250.1455478095.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BITRIX_SM_GUEST_ID=5913601; BITRIX_SM_LAST_VISIT=14.02.2016+22%3A35%3A57
    Code:
    HTTP/1.1 200 OK
Server: nginx/0.8.53
Date: Sun, 14 Feb 2016 21:00:37 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.18
P3P: policyref="/bitrix/p3p.xml", CP="NON DSP COR CUR ADM DEV PSA PSD OUR UNR BUS UNI COM NAV INT DEM STA"
X-Powered-CMS: Bitrix Site Manager (19b839d1031a99892d4b0429297bad16)
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Set-Cookie: PHPSESSID=42b7b0c8316a3bcf3380c31923f97ca8; path=/; domain=it-weekly.ru; HttpOnly
Set-Cookie: BITRIX_SM_GUEST_ID=5913601; expires=Wed, 08-Feb-2017 21:00:37 GMT; path=/; domain=it-weekly.ru
Set-Cookie: BITRIX_SM_LAST_VISIT=15.02.2016+00%3A00%3A37; expires=Wed, 08-Feb-2017 21:00:37 GMT; path=/; domain=it-weekly.ru
X-Bitrix-Ajax-Status: Authorize
Content-Length: 32286

    Затем уже пост
    Code:
    POST http://it-weekly.ru/personal/index.php?login=yes HTTP/1.1
Host: it-weekly.ru
Connection: keep-alive
Content-Length: 183
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://it-weekly.ru
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36
Content-Type: application/x-www-form-urlencoded
DNT: 1
Referer: http://it-weekly.ru/personal/
Accept-Encoding: gzip, deflate
Accept-Language: ru,en;q=0.8
Cookie: _ym_uid=1455478095228020382; BX_USER_ID=f9d2a4228ad5ae9bd08b29889bb3aaf6; _ym_isad=0; PHPSESSID=42b7b0c8316a3bcf3380c31923f97ca8; __utmt=1; BITRIX_SM_SOUND_LOGIN_PLAYED=Y; BITRIX_SM_GUEST_ID=5913601; BITRIX_SM_LAST_VISIT=15.02.2016+00%3A01%3A13; resolution=2560; __utma=24257250.2079285695.1455478095.1455478095.1455483642.2; __utmb=24257250.2.10.1455483642; __utmc=24257250; __utmz=24257250.1455478095.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

AUTH_FORM=Y&TYPE=AUTH&backurl=%2Fpersonal%2Findex.php&USER_LOGIN=qweqwe&USER_PASSWORD=qweqwe&Login=%D0%90%D0%B2%D1%82%D0%BE%D1%80%D0%B8%D0%B7%D0%BE%D0%B2%D0%B0%D1%82%D1%8C%D1%81%D1%8F
    Code:
    
HTTP/1.1 200 OK
Server: nginx/0.8.53
Date: Sun, 14 Feb 2016 21:01:21 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Powered-By: PHP/5.3.18
P3P: policyref="/bitrix/p3p.xml", CP="NON DSP COR CUR ADM DEV PSA PSD OUR UNR BUS UNI COM NAV INT DEM STA"
X-Powered-CMS: Bitrix Site Manager (19b839d1031a99892d4b0429297bad16)
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Set-Cookie: BITRIX_SM_SALE_UID=0; expires=Wed, 08-Feb-2017 21:01:21 GMT; path=/; domain=it-weekly.ru
Set-Cookie: BITRIX_SM_SALE_UID=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=it-weekly.ru
Set-Cookie: BITRIX_SM_LOGIN=qweqwe; expires=Mon, 18-Jan-2021 21:01:21 GMT; path=/; domain=it-weekly.ru
Set-Cookie: BITRIX_SM_GUEST_ID=5913601; expires=Wed, 08-Feb-2017 21:01:21 GMT; path=/; domain=it-weekly.ru
Set-Cookie: BITRIX_SM_LAST_VISIT=15.02.2016+00%3A01%3A21; expires=Wed, 08-Feb-2017 21:01:21 GMT; path=/; domain=it-weekly.ru
X-Bitrix-Ajax-Status: Authorize
Content-Length: 32923

    P.s. BX_USER_ID=f9d2a4228ad5ae9bd08b29889bb3aaf6 этота кука подгружается из папки /gxstat/ или как то так, но думаю без нее тоже будет гуд

    Set-Cookies парсь и авторизация гуд...


    [​IMG]
     
    #13 lifescore, 15 Feb 2016
  14. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    в том то и проблема что гет запрос даже без кук не проходит =/
     
    #14 MrCable, 15 Feb 2016
  15. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    быть такого не может..
    если через сокс, то снкчало коннект идет, если это не автоматически конечно все делаеться

    попробуй проксю нацепи, может ты в списке изгоя у них уже)
     
    #15 lifescore, 15 Feb 2016
  16. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Проксю нацепил, но ничего не поменялось, вот весь запрос, может где в нем накосячил ?!

    PHP:
    function curl($url){
           // $cookie_jar = tempnam('/tmp','cookie');
       
            $headers = array(
            'GET ' . $url . ' HTTP/1.0',
            'Accept: text/html, application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
            'Accept-Encoding: gzip, deflate',
            'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3',
            'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0',
            'Connection: keep-alive'
    );



            $ch = curl_init();
    curl_setopt ($ch , CURLOPT_URL , $url );
    curl_setopt ($ch , CURLOPT_USERAGENT , "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:44.0) Gecko/20100101 Firefox/44.0");
    curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 0);
    curl_setopt($ch, CURLOPT_PROXY, '198.99.224.134:80');
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt ($ch, CURLOPT_HEADER, 1);
    curl_setopt ($ch, CURLOPT_REFERER, 'https://www.google.com.ua');
    curl_setopt ($ch, CURLOPT_HTTPHEADER, $headers);
    //curl_setopt ($ch, CURLOPT_COOKIEFILE, "$cookie_jar");
    //curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookie_jar);
    curl_setopt ($ch , CURLOPT_RETURNTRANSFER , 1 );
    $file = curl_exec($ch);
            // remove the cookie jar
    //unlink($cookie_jar) or die("Can't unlink $cookie_jar");
            return $file;
      
       
    }
     
    #16 MrCable, 15 Feb 2016
  17. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    #17 lifescore, 16 Feb 2016
  18. Waki

    Waki Member

    Joined:
    9 Oct 2015
    Messages:
    54
    Likes Received:
    30
    Reputations:
    10
    Тебе не надо отправлять весь заголовок который был получен в фидлере, в твоем случае надо убрать в $headers
    Code:
    'GET ' . $url . ' HTTP/1.0',
     
    #18 Waki, 16 Feb 2016
    lifescore likes this.
  19. lifescore

    lifescore Elder - Старейшина

    Joined:
    27 Aug 2011
    Messages:
    640
    Likes Received:
    504
    Reputations:
    72
    так то да)) тупанули чет совсем..
     
    #19 lifescore, 16 Feb 2016
  20. MrCable

    MrCable New Member

    Joined:
    10 Feb 2016
    Messages:
    10
    Likes Received:
    1
    Reputations:
    0
    Я то убрал но не проходит все равно, сейчас почитаю что написано в статье выше, может где то ошибся я.
     
    #20 MrCable, 16 Feb 2016
(You must log in or sign up to post here.)
Page 1 of 2