What does “fetching by region is not available for SAM files” mean? Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern) Announcing the arrival of Valued Associate #679: Cesar Manara Unicorn Meta Zoo #1: Why another podcast?How to safely and efficiently convert subset of bam to fastq?How to you find read statistics for bam files with terminal mismatch?Pepsin digest (cleavage) does not work using RE?running a samtools command for multiple bam files from 1000 genomes projectSAM format: Does the BAM “Integer or numeric array” field no longer exist? Why?python does not quit when the input file is too bigconvert supplementary reads to primary in sam or bamI have 23andme text files and would like to convert to SAM/BAM formatRunning htseq-count over BAM filesAccess base aligned to particular reference position

Why doesn't SQL Optimizer use my constraint?

Amount of permutations on an NxNxN Rubik's Cube

Drawing without replacement: why the order of draw is irrelevant?

Sum letters are not two different

Why wasn't DOSKEY integrated with COMMAND.COM?

Trademark violation for app?

Can a new player join a group only when a new campaign starts?

How to write dimensions below a matrix

Localisation of Category

Why is it faster to reheat something than it is to cook it?

Why aren't air breathing engines used as small first stages?

Did Krishna say in Bhagavad Gita "I am in every living being"

Dating a Former Employee

Do I really need to have a message in a novel to appeal to readers?

An adverb for when you're not exaggerating

Export Xpubkey from Bitcoin Core

Is grep documentation about ignoring case wrong, since it doesn't ignore case in filenames?

Should I use a zero-interest credit card for a large one-time purchase?

How does light 'choose' between wave and particle behaviour?

How to compare two different files line by line in unix?

What is the font for "b" letter?

How do I use the new nonlinear finite element in Mathematica 12 for this equation?

Why should I vote and accept answers?

Chinese Seal on silk painting - what does it mean?



What does “fetching by region is not available for SAM files” mean?



Planned maintenance scheduled April 23, 2019 at 00:00UTC (8:00pm US/Eastern)
Announcing the arrival of Valued Associate #679: Cesar Manara
Unicorn Meta Zoo #1: Why another podcast?How to safely and efficiently convert subset of bam to fastq?How to you find read statistics for bam files with terminal mismatch?Pepsin digest (cleavage) does not work using RE?running a samtools command for multiple bam files from 1000 genomes projectSAM format: Does the BAM “Integer or numeric array” field no longer exist? Why?python does not quit when the input file is too bigconvert supplementary reads to primary in sam or bamI have 23andme text files and would like to convert to SAM/BAM formatRunning htseq-count over BAM filesAccess base aligned to particular reference position










5












$begingroup$


I am used to gzip/biopython solutions when dealing with sequencing data, but now I wish to switch to more elegant pysam. So I looked at the manual, but ran into quite bizarre troubles with the first couple of lines using my bam file



import pysam
samfile = pysam.AlignmentFile("3_Tms_1_mapped.bam", "rb")
for read in samfile.fetch('3_Tms_b3v08_scaf000159'):
print(read)
samfile.close()


returns ValueError: fetching by region is not available for SAM files. Well, the file is bam. I tried to google the error but the only hits I found are the lines in the source code of pysam that check if the file is bam/cram or sam, so somehow pysam thinks that my bam is a sam. How can I convince it otherwise? I have also noticed that the manual is for python 2.7, that's maybe where the problem comes from...



If you wish to reproduce my problem, here is my bam file.










share|improve this question











$endgroup$
















    5












    $begingroup$


    I am used to gzip/biopython solutions when dealing with sequencing data, but now I wish to switch to more elegant pysam. So I looked at the manual, but ran into quite bizarre troubles with the first couple of lines using my bam file



    import pysam
    samfile = pysam.AlignmentFile("3_Tms_1_mapped.bam", "rb")
    for read in samfile.fetch('3_Tms_b3v08_scaf000159'):
    print(read)
    samfile.close()


    returns ValueError: fetching by region is not available for SAM files. Well, the file is bam. I tried to google the error but the only hits I found are the lines in the source code of pysam that check if the file is bam/cram or sam, so somehow pysam thinks that my bam is a sam. How can I convince it otherwise? I have also noticed that the manual is for python 2.7, that's maybe where the problem comes from...



    If you wish to reproduce my problem, here is my bam file.










    share|improve this question











    $endgroup$














      5












      5








      5





      $begingroup$


      I am used to gzip/biopython solutions when dealing with sequencing data, but now I wish to switch to more elegant pysam. So I looked at the manual, but ran into quite bizarre troubles with the first couple of lines using my bam file



      import pysam
      samfile = pysam.AlignmentFile("3_Tms_1_mapped.bam", "rb")
      for read in samfile.fetch('3_Tms_b3v08_scaf000159'):
      print(read)
      samfile.close()


      returns ValueError: fetching by region is not available for SAM files. Well, the file is bam. I tried to google the error but the only hits I found are the lines in the source code of pysam that check if the file is bam/cram or sam, so somehow pysam thinks that my bam is a sam. How can I convince it otherwise? I have also noticed that the manual is for python 2.7, that's maybe where the problem comes from...



      If you wish to reproduce my problem, here is my bam file.










      share|improve this question











      $endgroup$




      I am used to gzip/biopython solutions when dealing with sequencing data, but now I wish to switch to more elegant pysam. So I looked at the manual, but ran into quite bizarre troubles with the first couple of lines using my bam file



      import pysam
      samfile = pysam.AlignmentFile("3_Tms_1_mapped.bam", "rb")
      for read in samfile.fetch('3_Tms_b3v08_scaf000159'):
      print(read)
      samfile.close()


      returns ValueError: fetching by region is not available for SAM files. Well, the file is bam. I tried to google the error but the only hits I found are the lines in the source code of pysam that check if the file is bam/cram or sam, so somehow pysam thinks that my bam is a sam. How can I convince it otherwise? I have also noticed that the manual is for python 2.7, that's maybe where the problem comes from...



      If you wish to reproduce my problem, here is my bam file.







      python bam pysam






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Apr 11 at 10:17









      terdon

      4,8202830




      4,8202830










      asked Apr 10 at 15:51









      Kamil S JaronKamil S Jaron

      2,957742




      2,957742




















          2 Answers
          2






          active

          oldest

          votes


















          5












          $begingroup$

          Your 3_Tms_1_mapped.bam file, despite its filename extension, is in fact a bgzipped SAM file. You can verify this using htsfile, which is a small utility packaged with HTSlib:



          $ htsfile 3_Tms_1_mapped.bam 
          3_Tms_1_mapped.bam: SAM version 1.3 BGZF-compressed sequence data


          (For files that really are in BAM format, it reports BAM version 1 compressed sequence data.)



          So the error message is accurate in this case.






          share|improve this answer











          $endgroup$












          • $begingroup$
            Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 21:52






          • 1




            $begingroup$
            No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
            $endgroup$
            – John Marshall
            Apr 10 at 21:58










          • $begingroup$
            I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
            $endgroup$
            – Kamil S Jaron
            Apr 11 at 7:05







          • 1




            $begingroup$
            @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
            $endgroup$
            – terdon
            Apr 11 at 8:10


















          4












          $begingroup$

          That isn't actually a bam file as John Marshall figured out. I am keeping the rest of my answer since it could be useful to someone else, but the issue here was that you had a compressed (bgzipped) sam file and not an actual bam file and that's why you were getting that error. When I sorted your file in preparation for indexing it, I converted to a bam which is why the rest of my answer worked.




          You don't have the index file for the bam file you're using. I used this script on the file you linked to (changing the name so that it corresponds to the right file and a contig in that file):



          #!/usr/bin/env python3

          import pysam
          samfile = pysam.AlignmentFile("3_Tce_1_mapped.bam", "rb")
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          print(read)
          samfile.close()


          The directory I ran it in had:



          $ ls 3*
          3_Tce_1_mapped.bam


          And I got the error you described:



          $ foo.py
          Traceback (most recent call last):
          File "/home/terdon/scripts/foo.py", line 5, in <module>
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          File "pysam/libcalignmentfile.pyx", line 1107, in pysam.libcalignmentfile.AlignmentFile.fetch
          ValueError: fetching by region is not available for SAM files


          However, indexing the bam file fixed it:



          $ samtools sort 3_Tce_1_mapped.bam > 3_Tce_1_mapped.sorted.bam 
          $
          mv 3_Tce_1_mapped.sorted.bam 3_Tce_1_mapped.bam
          $ samtools index 3_Tce_1_mapped.bam
          $
          ls 3*
          3_Tce_1_mapped.bam 3_Tce_1_mapped.bam.bai

          $ foo.py | wc
          227 2724 16725


          So just sort and index your files before attempting to seek in them. Which makes sense since the index's job is primarily to allow seeking.






          share|improve this answer











          $endgroup$












          • $begingroup$
            I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 18:37










          • $begingroup$
            Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:40










          • $begingroup$
            @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
            $endgroup$
            – terdon
            Apr 10 at 18:40






          • 1




            $begingroup$
            @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:41










          • $begingroup$
            @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
            $endgroup$
            – terdon
            Apr 10 at 18:42











          Your Answer








          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "676"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: false,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: null,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fbioinformatics.stackexchange.com%2fquestions%2f7436%2fwhat-does-fetching-by-region-is-not-available-for-sam-files-mean%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          2 Answers
          2






          active

          oldest

          votes








          2 Answers
          2






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes









          5












          $begingroup$

          Your 3_Tms_1_mapped.bam file, despite its filename extension, is in fact a bgzipped SAM file. You can verify this using htsfile, which is a small utility packaged with HTSlib:



          $ htsfile 3_Tms_1_mapped.bam 
          3_Tms_1_mapped.bam: SAM version 1.3 BGZF-compressed sequence data


          (For files that really are in BAM format, it reports BAM version 1 compressed sequence data.)



          So the error message is accurate in this case.






          share|improve this answer











          $endgroup$












          • $begingroup$
            Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 21:52






          • 1




            $begingroup$
            No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
            $endgroup$
            – John Marshall
            Apr 10 at 21:58










          • $begingroup$
            I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
            $endgroup$
            – Kamil S Jaron
            Apr 11 at 7:05







          • 1




            $begingroup$
            @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
            $endgroup$
            – terdon
            Apr 11 at 8:10















          5












          $begingroup$

          Your 3_Tms_1_mapped.bam file, despite its filename extension, is in fact a bgzipped SAM file. You can verify this using htsfile, which is a small utility packaged with HTSlib:



          $ htsfile 3_Tms_1_mapped.bam 
          3_Tms_1_mapped.bam: SAM version 1.3 BGZF-compressed sequence data


          (For files that really are in BAM format, it reports BAM version 1 compressed sequence data.)



          So the error message is accurate in this case.






          share|improve this answer











          $endgroup$












          • $begingroup$
            Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 21:52






          • 1




            $begingroup$
            No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
            $endgroup$
            – John Marshall
            Apr 10 at 21:58










          • $begingroup$
            I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
            $endgroup$
            – Kamil S Jaron
            Apr 11 at 7:05







          • 1




            $begingroup$
            @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
            $endgroup$
            – terdon
            Apr 11 at 8:10













          5












          5








          5





          $begingroup$

          Your 3_Tms_1_mapped.bam file, despite its filename extension, is in fact a bgzipped SAM file. You can verify this using htsfile, which is a small utility packaged with HTSlib:



          $ htsfile 3_Tms_1_mapped.bam 
          3_Tms_1_mapped.bam: SAM version 1.3 BGZF-compressed sequence data


          (For files that really are in BAM format, it reports BAM version 1 compressed sequence data.)



          So the error message is accurate in this case.






          share|improve this answer











          $endgroup$



          Your 3_Tms_1_mapped.bam file, despite its filename extension, is in fact a bgzipped SAM file. You can verify this using htsfile, which is a small utility packaged with HTSlib:



          $ htsfile 3_Tms_1_mapped.bam 
          3_Tms_1_mapped.bam: SAM version 1.3 BGZF-compressed sequence data


          (For files that really are in BAM format, it reports BAM version 1 compressed sequence data.)



          So the error message is accurate in this case.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Apr 11 at 6:36

























          answered Apr 10 at 20:43









          John MarshallJohn Marshall

          56826




          56826











          • $begingroup$
            Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 21:52






          • 1




            $begingroup$
            No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
            $endgroup$
            – John Marshall
            Apr 10 at 21:58










          • $begingroup$
            I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
            $endgroup$
            – Kamil S Jaron
            Apr 11 at 7:05







          • 1




            $begingroup$
            @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
            $endgroup$
            – terdon
            Apr 11 at 8:10
















          • $begingroup$
            Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 21:52






          • 1




            $begingroup$
            No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
            $endgroup$
            – John Marshall
            Apr 10 at 21:58










          • $begingroup$
            I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
            $endgroup$
            – Kamil S Jaron
            Apr 11 at 7:05







          • 1




            $begingroup$
            @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
            $endgroup$
            – terdon
            Apr 11 at 8:10















          $begingroup$
          Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
          $endgroup$
          – Kamil S Jaron
          Apr 10 at 21:52




          $begingroup$
          Hmm, does it mean that this link is wrong? biopython.org/DIST/docs/api/Bio.bgzf-module.html I sort of thought that it's the same thing...
          $endgroup$
          – Kamil S Jaron
          Apr 10 at 21:52




          1




          1




          $begingroup$
          No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
          $endgroup$
          – John Marshall
          Apr 10 at 21:58




          $begingroup$
          No. (Was there a particular part of that you think might be wrong?) But there is a difference between BGZF-compressing the plain text format, and a BAM file (whose decompressed underlying stream is a bespoke binary format that is different from the plain SAM text).
          $endgroup$
          – John Marshall
          Apr 10 at 21:58












          $begingroup$
          I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
          $endgroup$
          – Kamil S Jaron
          Apr 11 at 7:05





          $begingroup$
          I think I get it now. I got confused by the aims to this module, they write "In addition to being required for random access to and writing of BAM files", so I got an impression that I can simply use this module to write BAM files. This was actually really helpful answer, thanks!
          $endgroup$
          – Kamil S Jaron
          Apr 11 at 7:05





          1




          1




          $begingroup$
          @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
          $endgroup$
          – terdon
          Apr 11 at 8:10




          $begingroup$
          @KamilSJaron you might want to consider accepting this answer instead of mine since this was the real issue.
          $endgroup$
          – terdon
          Apr 11 at 8:10











          4












          $begingroup$

          That isn't actually a bam file as John Marshall figured out. I am keeping the rest of my answer since it could be useful to someone else, but the issue here was that you had a compressed (bgzipped) sam file and not an actual bam file and that's why you were getting that error. When I sorted your file in preparation for indexing it, I converted to a bam which is why the rest of my answer worked.




          You don't have the index file for the bam file you're using. I used this script on the file you linked to (changing the name so that it corresponds to the right file and a contig in that file):



          #!/usr/bin/env python3

          import pysam
          samfile = pysam.AlignmentFile("3_Tce_1_mapped.bam", "rb")
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          print(read)
          samfile.close()


          The directory I ran it in had:



          $ ls 3*
          3_Tce_1_mapped.bam


          And I got the error you described:



          $ foo.py
          Traceback (most recent call last):
          File "/home/terdon/scripts/foo.py", line 5, in <module>
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          File "pysam/libcalignmentfile.pyx", line 1107, in pysam.libcalignmentfile.AlignmentFile.fetch
          ValueError: fetching by region is not available for SAM files


          However, indexing the bam file fixed it:



          $ samtools sort 3_Tce_1_mapped.bam > 3_Tce_1_mapped.sorted.bam 
          $
          mv 3_Tce_1_mapped.sorted.bam 3_Tce_1_mapped.bam
          $ samtools index 3_Tce_1_mapped.bam
          $
          ls 3*
          3_Tce_1_mapped.bam 3_Tce_1_mapped.bam.bai

          $ foo.py | wc
          227 2724 16725


          So just sort and index your files before attempting to seek in them. Which makes sense since the index's job is primarily to allow seeking.






          share|improve this answer











          $endgroup$












          • $begingroup$
            I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 18:37










          • $begingroup$
            Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:40










          • $begingroup$
            @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
            $endgroup$
            – terdon
            Apr 10 at 18:40






          • 1




            $begingroup$
            @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:41










          • $begingroup$
            @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
            $endgroup$
            – terdon
            Apr 10 at 18:42















          4












          $begingroup$

          That isn't actually a bam file as John Marshall figured out. I am keeping the rest of my answer since it could be useful to someone else, but the issue here was that you had a compressed (bgzipped) sam file and not an actual bam file and that's why you were getting that error. When I sorted your file in preparation for indexing it, I converted to a bam which is why the rest of my answer worked.




          You don't have the index file for the bam file you're using. I used this script on the file you linked to (changing the name so that it corresponds to the right file and a contig in that file):



          #!/usr/bin/env python3

          import pysam
          samfile = pysam.AlignmentFile("3_Tce_1_mapped.bam", "rb")
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          print(read)
          samfile.close()


          The directory I ran it in had:



          $ ls 3*
          3_Tce_1_mapped.bam


          And I got the error you described:



          $ foo.py
          Traceback (most recent call last):
          File "/home/terdon/scripts/foo.py", line 5, in <module>
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          File "pysam/libcalignmentfile.pyx", line 1107, in pysam.libcalignmentfile.AlignmentFile.fetch
          ValueError: fetching by region is not available for SAM files


          However, indexing the bam file fixed it:



          $ samtools sort 3_Tce_1_mapped.bam > 3_Tce_1_mapped.sorted.bam 
          $
          mv 3_Tce_1_mapped.sorted.bam 3_Tce_1_mapped.bam
          $ samtools index 3_Tce_1_mapped.bam
          $
          ls 3*
          3_Tce_1_mapped.bam 3_Tce_1_mapped.bam.bai

          $ foo.py | wc
          227 2724 16725


          So just sort and index your files before attempting to seek in them. Which makes sense since the index's job is primarily to allow seeking.






          share|improve this answer











          $endgroup$












          • $begingroup$
            I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 18:37










          • $begingroup$
            Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:40










          • $begingroup$
            @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
            $endgroup$
            – terdon
            Apr 10 at 18:40






          • 1




            $begingroup$
            @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:41










          • $begingroup$
            @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
            $endgroup$
            – terdon
            Apr 10 at 18:42













          4












          4








          4





          $begingroup$

          That isn't actually a bam file as John Marshall figured out. I am keeping the rest of my answer since it could be useful to someone else, but the issue here was that you had a compressed (bgzipped) sam file and not an actual bam file and that's why you were getting that error. When I sorted your file in preparation for indexing it, I converted to a bam which is why the rest of my answer worked.




          You don't have the index file for the bam file you're using. I used this script on the file you linked to (changing the name so that it corresponds to the right file and a contig in that file):



          #!/usr/bin/env python3

          import pysam
          samfile = pysam.AlignmentFile("3_Tce_1_mapped.bam", "rb")
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          print(read)
          samfile.close()


          The directory I ran it in had:



          $ ls 3*
          3_Tce_1_mapped.bam


          And I got the error you described:



          $ foo.py
          Traceback (most recent call last):
          File "/home/terdon/scripts/foo.py", line 5, in <module>
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          File "pysam/libcalignmentfile.pyx", line 1107, in pysam.libcalignmentfile.AlignmentFile.fetch
          ValueError: fetching by region is not available for SAM files


          However, indexing the bam file fixed it:



          $ samtools sort 3_Tce_1_mapped.bam > 3_Tce_1_mapped.sorted.bam 
          $
          mv 3_Tce_1_mapped.sorted.bam 3_Tce_1_mapped.bam
          $ samtools index 3_Tce_1_mapped.bam
          $
          ls 3*
          3_Tce_1_mapped.bam 3_Tce_1_mapped.bam.bai

          $ foo.py | wc
          227 2724 16725


          So just sort and index your files before attempting to seek in them. Which makes sense since the index's job is primarily to allow seeking.






          share|improve this answer











          $endgroup$



          That isn't actually a bam file as John Marshall figured out. I am keeping the rest of my answer since it could be useful to someone else, but the issue here was that you had a compressed (bgzipped) sam file and not an actual bam file and that's why you were getting that error. When I sorted your file in preparation for indexing it, I converted to a bam which is why the rest of my answer worked.




          You don't have the index file for the bam file you're using. I used this script on the file you linked to (changing the name so that it corresponds to the right file and a contig in that file):



          #!/usr/bin/env python3

          import pysam
          samfile = pysam.AlignmentFile("3_Tce_1_mapped.bam", "rb")
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          print(read)
          samfile.close()


          The directory I ran it in had:



          $ ls 3*
          3_Tce_1_mapped.bam


          And I got the error you described:



          $ foo.py
          Traceback (most recent call last):
          File "/home/terdon/scripts/foo.py", line 5, in <module>
          for read in samfile.fetch('3_Tce_b3v08_scaf005149'):
          File "pysam/libcalignmentfile.pyx", line 1107, in pysam.libcalignmentfile.AlignmentFile.fetch
          ValueError: fetching by region is not available for SAM files


          However, indexing the bam file fixed it:



          $ samtools sort 3_Tce_1_mapped.bam > 3_Tce_1_mapped.sorted.bam 
          $
          mv 3_Tce_1_mapped.sorted.bam 3_Tce_1_mapped.bam
          $ samtools index 3_Tce_1_mapped.bam
          $
          ls 3*
          3_Tce_1_mapped.bam 3_Tce_1_mapped.bam.bai

          $ foo.py | wc
          227 2724 16725


          So just sort and index your files before attempting to seek in them. Which makes sense since the index's job is primarily to allow seeking.







          share|improve this answer














          share|improve this answer



          share|improve this answer








          edited Apr 11 at 9:54

























          answered Apr 10 at 16:10









          terdonterdon

          4,8202830




          4,8202830











          • $begingroup$
            I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 18:37










          • $begingroup$
            Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:40










          • $begingroup$
            @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
            $endgroup$
            – terdon
            Apr 10 at 18:40






          • 1




            $begingroup$
            @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:41










          • $begingroup$
            @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
            $endgroup$
            – terdon
            Apr 10 at 18:42
















          • $begingroup$
            I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
            $endgroup$
            – Kamil S Jaron
            Apr 10 at 18:37










          • $begingroup$
            Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:40










          • $begingroup$
            @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
            $endgroup$
            – terdon
            Apr 10 at 18:40






          • 1




            $begingroup$
            @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
            $endgroup$
            – Daniel Standage
            Apr 10 at 18:41










          • $begingroup$
            @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
            $endgroup$
            – terdon
            Apr 10 at 18:42















          $begingroup$
          I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
          $endgroup$
          – Kamil S Jaron
          Apr 10 at 18:37




          $begingroup$
          I was thinking about indexing, but then I thought that if that would be the problem pysam would complain about sorting/indexing, not about file being sam. Do yo think it would be worth opening them a ticket to make the error message bit more meaningful?
          $endgroup$
          – Kamil S Jaron
          Apr 10 at 18:37












          $begingroup$
          Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
          $endgroup$
          – Daniel Standage
          Apr 10 at 18:40




          $begingroup$
          Seeking, i.e. random access, requires an index. Note that this is not the only access pattern in pysam. You can access reads in a streaming fashion (for read in samfile:) without sorting, and you can access read pileups base-by-base (for column in samfile.pileup('chr1')), which does require an index if I understand correctly.
          $endgroup$
          – Daniel Standage
          Apr 10 at 18:40












          $begingroup$
          @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
          $endgroup$
          – terdon
          Apr 10 at 18:40




          $begingroup$
          @KamilSJaron probably, yes. It obviously caused you some pain, and I'm sure you won't be alone. I only thought of this because I have seen similar cryptic errors and was wondering how it could seek without an index anyway.
          $endgroup$
          – terdon
          Apr 10 at 18:40




          1




          1




          $begingroup$
          @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
          $endgroup$
          – Daniel Standage
          Apr 10 at 18:41




          $begingroup$
          @KamilSJaron Yes, I think it's an unhelpful error message, and it has caused me confusion in the past as well. If you open a ticket on Github I'd be happy to "pile on." :-)
          $endgroup$
          – Daniel Standage
          Apr 10 at 18:41












          $begingroup$
          @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
          $endgroup$
          – terdon
          Apr 10 at 18:42




          $begingroup$
          @DanielStandage I am this close to suspending you for that horrible pun! I would too if I could stop chuckling for long enough!
          $endgroup$
          – terdon
          Apr 10 at 18:42

















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Bioinformatics Stack Exchange!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          Use MathJax to format equations. MathJax reference.


          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fbioinformatics.stackexchange.com%2fquestions%2f7436%2fwhat-does-fetching-by-region-is-not-available-for-sam-files-mean%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          Romeo and Juliet ContentsCharactersSynopsisSourcesDate and textThemes and motifsCriticism and interpretationLegacyScene by sceneSee alsoNotes and referencesSourcesExternal linksNavigation menu"Consumer Price Index (estimate) 1800–"10.2307/28710160037-3222287101610.1093/res/II.5.31910.2307/45967845967810.2307/2869925286992510.1525/jams.1982.35.3.03a00050"Dada Masilo: South African dancer who breaks the rules"10.1093/res/os-XV.57.1610.2307/28680942868094"Sweet Sorrow: Mann-Korman's Romeo and Juliet Closes Sept. 5 at MN's Ordway"the original10.2307/45957745957710.1017/CCOL0521570476.009"Ram Leela box office collections hit massive Rs 100 crore, pulverises prediction"Archived"Broadway Revival of Romeo and Juliet, Starring Orlando Bloom and Condola Rashad, Will Close Dec. 8"Archived10.1075/jhp.7.1.04hon"Wherefore art thou, Romeo? To make us laugh at Navy Pier"the original10.1093/gmo/9781561592630.article.O006772"Ram-leela Review Roundup: Critics Hail Film as Best Adaptation of Romeo and Juliet"Archived10.2307/31946310047-77293194631"Romeo and Juliet get Twitter treatment""Juliet's Nurse by Lois Leveen""Romeo and Juliet: Orlando Bloom's Broadway Debut Released in Theaters for Valentine's Day"Archived"Romeo and Juliet Has No Balcony"10.1093/gmo/9781561592630.article.O00778110.2307/2867423286742310.1076/enst.82.2.115.959510.1080/00138380601042675"A plague o' both your houses: error in GCSE exam paper forces apology""Juliet of the Five O'Clock Shadow, and Other Wonders"10.2307/33912430027-4321339124310.2307/28487440038-7134284874410.2307/29123140149-661129123144728341M"Weekender Guide: Shakespeare on The Drive""balcony"UK public library membership"romeo"UK public library membership10.1017/CCOL9780521844291"Post-Zionist Critique on Israel and the Palestinians Part III: Popular Culture"10.2307/25379071533-86140377-919X2537907"Capulets and Montagues: UK exam board admit mixing names up in Romeo and Juliet paper"Istoria Novellamente Ritrovata di Due Nobili Amanti2027/mdp.390150822329610820-750X"GCSE exam error: Board accidentally rewrites Shakespeare"10.2307/29176390149-66112917639"Exam board apologises after error in English GCSE paper which confused characters in Shakespeare's Romeo and Juliet""From Mariotto and Ganozza to Romeo and Guilietta: Metamorphoses of a Renaissance Tale"10.2307/37323537323510.2307/2867455286745510.2307/28678912867891"10 Questions for Taylor Swift"10.2307/28680922868092"Haymarket Theatre""The Zeffirelli Way: Revealing Talk by Florentine Director""Michael Smuin: 1938-2007 / Prolific dance director had showy career"The Life and Art of Edwin BoothRomeo and JulietRomeo and JulietRomeo and JulietRomeo and JulietEasy Read Romeo and JulietRomeo and Julieteeecb12003684p(data)4099369-3n8211610759dbe00d-a9e2-41a3-b2c1-977dd692899302814385X313670221313670221

          Creating closest line along the point''s azimuth using PostgreSQL Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) Announcing the arrival of Valued Associate #679: Cesar Manara Unicorn Meta Zoo #1: Why another podcast?Drawing line between points at specific distance in PostGIS?How to efficiently find the closest point over the dateline?How to find the nearest point by using PostGIS function?PostGIS nearest point with LATERAL JOIN in PostgreSQL 9.3+Creating a table and inserting selected streets using plpgsql functionsCreating a table that stores Distances and other columnSaving select query results (year wise) from PostgreSQL/PostGIS to text filesWhat is the information behind this geometry?How to give start and end vertex ids dynamically in pgr_dijkstra?Point to Polygon nearest distance DS_distance is not using geography index & knn <-> or <#> does not give result in orderLine to point conversion with start point and end point detection?

          Crop image to path created in TikZ? Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern)Crop an inserted image?TikZ pictures does not appear in posterImage behind and beyond crop marks?Tikz picture as large as possible on A4 PageTransparency vs image compression dilemmaHow to crop background from image automatically?Image does not cropTikzexternal capturing crop marks when externalizing pgfplots?How to include image path that contains a dollar signCrop image with left size given